System and method of measuring distances related to an object utilizing ancillary objects

ABSTRACT

A system and method for measuring distances related to a target object depicted in an image and the construction and delivery of supplemental window materials for fenestration. A captured digital image is obtained containing a scene with a target object whose dimension is to be measured. The digital image may contain a target object dimension identified by one or more ancillary objects and a reference object in the same or different planes. Image processing is performed to find the reference object using known fiducial patterns printed on the reference object, metadata supplied by a user and/or by the detection of colored papers in the scene of the captured image. Adhering objects aid in keeping the reference object applied to an item in the scene such as a wall, while contrast objects aid the image processing to locate the reference object in low contrast reference object/background situations. Once located and measured, the reference object is used to calculate a pixel scale factor used to measure the target object dimensions. Target object dimensions are provided to an automated or semi-automated measurement process, design and manufacturing system such that customized parts are provided to end users.

REFERENCE TO PRIORITY APPLICATION

This application is a continuation of U.S. application Ser. No.14/865,250, filed Sep. 25, 2015, entitled “System and Method ofMeasuring Distances Related to an Object Utilizing Ancillary Objects,”which is a continuation-in-part of U.S. application Ser. No. 14/320,973,filed Jul. 1, 2014, entitled “System and Method of Measuring DistancesRelated to an Object,” now U.S. Pat. No. 9,230,339, which is acontinuation-in-part of U.S. application Ser. No. 13/735,449, filed Jan.7, 2013, entitled “System and Method of Measuring Distances Related toan Object,” now U.S. Pat. No. 8,923,650, all of which are incorporatedherein by reference in their entirety.

FIELD OF THE INVENTION

The present invention relates generally to image processing and inparticular to a system and method for measuring the distances related toa target object depicted in an image utilizing ancillary objects and theconstruction and delivery of supplemental window materials forfenestration.

BACKGROUND OF THE INVENTION

In recognition of the ecological and cost impact of fossil fuels andother conventional energy sources, significant effort has been expendedin developing methods for more efficient use of such energy sources. Animportant area of energy use for which greater energy efficiency isneeded is the heating and cooling of spaces in which human activity isdesired. Many approaches have been developed to decrease the amount heattransfer through the shell of such spaces. One of the most active andimportant areas of activity is the transfer of energy throughfenestration where the activity has included use of window films orinserts, increasing the number of window glazings per opening and windowtreatments such as drapes, blinds, etc. While these approaches haveshown considerable improvement in building energy efficiency,significant problems prevent more widespread and effective utilization.

Several problems exist in the approaches to minimizing heat transferthrough fenestration. In particular for existing windows, it isdesirable to maintain the optical transparency of the window, operationof the window treatments (e.g., blinds) and windows and the aestheticsof the interior view of the window while providing thermal insulation.Furthermore, reuse of the insulating materials is highly desirable sothat new materials do not need to be purchased each season. When addingsupplemental window elements such as films, film support elements andwindow treatments, ease of installation (including measurement andfabrication), reusability and storage and aesthetics during and afteruse are very important while obtaining the thermal and radiationinsulation desired. With window films intended for creating anadditional “dead air” space adjacent to the window as well as windowtreatments, accurate measurement of the film dimensions is necessary,often requiring the assistance of a professional with the associatedadded cost and time. Other window films, such as tints, infrared orultraviolet reflective or absorbing films, or low-e films, adheredirectly to the windowpane and have similar issues. Additionally, withthe wide acceptance of mobile device applications that enable windowtreatment aesthetic choices to be made using images, it is desirable toadd image based measurement capability to such applications.

SUMMARY OF THE INVENTION

The present invention is a system and method for measuring the distancesrelated to an object depicted in an image and the construction anddelivery of supplemental window materials for fenestration. Oneembodiment of the present invention provides a method of photogrammetricmeasurement in which a digital image is obtained that contains a targetobject dimension and a reference object dimension in substantially thesame plane or line.

Another embodiment of the present invention provides a method ofphotogrammetric measurement in which a digital image is obtained thatcontains a target object dimension identified by an ancillary object anda reference object dimension in different planes. Automation ofembodiments of the present invention is facilitated by using fiducialpatterns on reference and optional ancillary objects that are recognizedby an image analysis algorithm.

A further embodiment of the present invention provides use of digitalimage processing thresholding methods with images of fenestration inwhich the background area of the transparent portion of the fenestrationhas contrast relative to the fenestration components visible in theimage adjacent to the transparent portion of the fenestration. In eachembodiment, a digital image undergoes digital image processing toprovide improved measurement capability.

In embodiments of the present invention, information regarding a targetobject, such as fenestration, and its immediate surroundings is providedto an automated or semi-automated measurement process, design andmanufacturing system such that customized parts are provided to endusers. In one method of the present invention, a digital image isobtained that contains at least a portion of an observable constraintdimension to which a customized part is to conform wherein the digitalimage contains a reference object having a reference dimension andcalculating a constraint dimension from the digital image based on areference dimension. The custom part is then designed and manufacturedbased on a calculated constraint dimension.

Another embodiment of the present invention provides an improvedinformation gathering method and data extraction where the measurementsused to design custom parts that meet the needs of the fenestration anduser are obtained from photographic information, e.g., digital images.The customized parts may include materials that provide for thermalinsulation, emissivity control, tinting, window treatments or mountingsupport for such materials.

The advantages of the system and method of measuring distancesassociated with fixed buildings, mobile homes, travel trailers and otherhabitations include the following.

The ease of specification of the supplemental element is improved forthe end user. The involvement of the end user in specifying, fabricatingand installing the supplemental element is minimized. The end user'sinvolvement is relatively easy to perform so the end user does notrequire a professional and requires minimal time commitment. Further,when automated measurements are employed using image processing andmetadata, such measurements may be easily associated and maintained withthe specific fenestration location.

The accuracy of the automated measurement is relatively high. Thisrelates to ease of use and removing the potential for end user errorfrom the process. Utilization of easily obtained and ubiquitous objects,apparatus and materials in an automated process allows the process toprovide accurate measurement of object dimensions. This is important forend user satisfaction and minimizing return of product. In addition,measurement accuracy allows for reduced cost and waste as well as easeof use.

The invention includes a capability for visual confirmation of designedparts and remote or customized support of end user installation. Thisrelates to the ease with which a design may be confirmed by the end userprior to fabrication. Since the end user and service provider orfabricator may view the same image easily, any necessary correction tothe design prior to fabrication is facilitated by the use of a digitalimage. In addition, the digital image may be used as part of remoteinstallation support or customized media that may be used by the enduser for each installation. This enhances end user satisfaction with thedelivered product and minimizes waste of time and materials due todesign error.

There is thus provided in accordance with the invention, a method ofestimating at least one dimension of a target object within a digitalimage which includes a reference object and one or more pane ancillaryobjects for aiding in determining the dimension of the target object,the method comprising obtaining a digital image containing the targetobject, reference object and one or more pane ancillary objects,locating the reference object in the digital image, calculating a pixelscale factor based on both measured and known dimensions of thereference object, locating the one or more pane ancillary objects and acorner or edge thereof in the digital image, locating the target objectin the digital image utilizing the corner or edge of one or more paneancillary objects, and calculating the at least one dimension of thetarget object in accordance with the pixel scale factor.

There is also provided in accordance with the invention, a method ofestimating at least one dimension of a non-transparent target objectcaptured within a digital image which includes a reference object foraiding in determining the dimension of the target object, the methodcomprising obtaining a digital image containing the target object,reference object, one or more adhering objects for affixing thereference object in substantially the same plane as the target object,and one or more contrast providing objects for providing backgroundcontrast between the reference object and the background adjacent to theedges of the reference object, locating the reference object in thedigital image aided by at least one of (a) a known fiducial patternprinted on the reference object, (b) metadata indicating the approximatelocation of either the reference object or the one or more adheringobjects, and (c) the location, color and/or visible shape of the one ormore contrast providing objects, calculating a pixel scale factor basedon both measured and known dimensions of the reference object, locatingthe target object in the digital image, and calculating the at least onedimension of the target object in accordance with the pixel scalefactor.

There is further provided in accordance with the invention, a method ofestimating at least one dimension of a target object within a digitalimage which includes a reference object and one or more ancillaryobjects for aiding in determining the dimension of the target object,the method comprising obtaining on a mobile device a digital imagecontaining the target object, reference object and one or more ancillaryobjects, transmitting the digital image to a server in communicationwith the mobile device, and receiving from the server at least onedimension of the target object, the dimension obtained by locating thereference object in the digital image, calculating a pixel scale factorbased on both measured and known dimensions of the reference object,locating the one or more ancillary objects and a corner or edge thereofin the digital image, locating the target object in the digital imageutilizing the corner or edge of one or more ancillary objects, andcalculating the at least one dimension of the target object inaccordance with the pixel scale factor.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIG. 1 is a block diagram illustrating an example computer processingsystem adapted to implement the measurement and image processingmechanism of the present invention;

FIG. 2 is a high level block diagram illustrating an exampletablet/mobile device incorporating the measurement and image processingmechanism of the present invention;

FIG. 3 is a block diagram illustrating an example room in which an enduser obtains a digital image of sample window;

FIG. 4 is a block diagram illustrating an example network showing thedata flow between fabricator, designer, service provider and end user;

FIG. 5 is a diagram illustrating an example window with reference objectand one or more ancillary objects;

FIG. 6 is a diagram illustrating the volume of space an end user must bein when acquiring the digital image of the window;

FIGS. 7A and 7B are flow diagrams illustrating an example overallworkflow between the end user and service provider;

FIGS. 8A and 8B are flow diagrams illustrating an example triageresolution imaging method;

FIG. 9 is a flow diagram illustrating an example method for determiningreference object dimensions;

FIG. 10 is a flow diagram illustrating a method for finding an ancillaryobject;

FIG. 11 is a flow diagram illustrating a method for finding a collectionof ancillary objects;

FIG. 12 is a flow diagram illustrating a method for finding windowpanes;

FIGS. 13A and 13B are flow diagrams illustrating a method fordetermining a perspective transform;

FIG. 14 is an illustration of destination points of a perspectivetransform;

FIG. 15 is a flow diagram illustrating an example preparation formeasurement method;

FIG. 16 is a flow diagram illustrating an example measurement method;

FIG. 17 is a flow diagram illustrating an example method for calculatingthe target object dimension substantially in the reference object plane;

FIG. 18 is a flow diagram illustrating an example method for calculatingthe target object dimension offset from the reference object plane;

FIG. 19 is a diagram illustrating orthographic and top views of offsetplane measurements;

FIG. 20 is a diagram illustrating parameters for camera locationdetermination;

FIG. 21 is a diagram illustrating an example window with referenceobject, adhering objects and pane ancillary objects;

FIG. 22 is a diagram illustrating an example window with referenceobject, adhering objects, pane ancillary objects and sealing interfaceancillary objects;

FIG. 23 is a diagram illustrating an example window with referenceobject on the frame portion of the window, adhering objects, contrastproviding objects, ancillary objects and sealing interface ancillaryobjects;

FIG. 24 is a diagram illustrating an example window with referenceobject on a non-transparent target object, adhering objects, contrastproviding objects, ancillary objects and sealing interface ancillaryobjects;

FIG. 25 is a flow diagram illustrating an example method of findingancillary objects;

FIG. 26 is a flow diagram illustrating an example method of determiningperspective transform; and

FIG. 27 is a flow diagram illustrating an example method of calculatingthe target object dimension.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a system and method for the measurementof distances related to a target object depicted in an image and theconstruction and delivery of supplemental materials and parts forfenestration. One embodiment of the invention includes a method ofphotogrammetric measurement in which a digital image is obtained thatcontains a target object dimension and a reference object dimension insubstantially the same plane or line. The digital image then undergoesdigital image processing to provide improved measurement capability. Inembodiments of the present invention, information regarding a targetobject, such as fenestration, and its immediate surroundings is providedto an automated or semi-automated measurement process, design andmanufacturing system such that customized materials and parts areprovided to end users.

In one method of the present invention, a digital image is obtained thatcontains at least a portion of an observable constraint dimension towhich a customized part conforms wherein the digital image contains areference object having a reference dimension. A constraint dimension isthen calculated from the digital image based on a reference dimension.The custom part is then designed and manufactured based on a calculatedconstraint dimension.

As will be appreciated by one skilled in the art, one or moreembodiments of the present invention may be embodied as a system,method, computer program product or any combination thereof.Accordingly, the present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,the present invention may take the form of a computer program productembodied in any tangible medium of expression having computer usableprogram code embodied in the medium.

The invention or portions thereof may be described in the generalcontext of computer-executable instructions, such as program modules,being executed by a computer. Generally, program modules includeroutines, programs, objects, components, data structures, etc., thatperform particular tasks or implement particular abstract data types.The invention may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

Any combination of one or more computer usable or computer readablemedium(s) may be utilized. The computer-usable or computer-readablemedium may be, for example but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatusor device. More specific examples (a non-exhaustive list) of thecomputer-readable medium would include the following: an electricalconnection having one or more wires, a portable computer diskette, ahard disk, a random access memory (RAM), a read-only memory (ROM), anerasable programmable read-only memory (EPROM or flash memory), anoptical fiber, a portable compact disc read-only memory (CDROM), opticalstorage device or a magnetic storage device. Note that thecomputer-usable or computer-readable medium could even be paper oranother suitable medium upon which the program is printed, as theprogram can be electronically captured, via, for instance, opticalscanning of the paper or other medium, then compiled, interpreted, orotherwise processed in a suitable manner, if necessary, and then storedin a computer memory. In the context of this document, a computer-usableor computer-readable medium may be any medium that can contain or storethe program for use by or in connection with the instruction executionsystem, apparatus, or device.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object oriented programming language such asJava, Smalltalk, Swift, C++, C# or the like and conventional proceduralprogramming languages, such as the C programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

The present invention is described below with reference to flowchartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products according to embodiments of the invention. Itwill be understood that each block of the flowchart illustrations and/orblock diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented or supported bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide processes for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

The invention is operational with numerous general purpose or specialpurpose computing system environments or configurations. Examples ofwell-known computing systems, environments, and/or configurations thatmay be suitable for use with the invention include, but are not limitedto, personal computers, server computers, cloud computing, hand-held orlaptop devices, multiprocessor systems, microprocessor, microcontrolleror microcomputer based systems, set top boxes, programmable consumerelectronics, ASIC or FPGA core, DSP core, network PCs, minicomputers,mainframe computers, distributed computing environments that include anyof the above systems or devices, and the like.

A block diagram illustrating an example computer processing systemadapted to implement the distance measurement and image processingmechanism of the present invention is shown in FIG. 1. The exemplarycomputer processing system, generally referenced 10, for implementingthe invention comprises a general purpose computing device 11. Computingdevice 11 comprises central processing unit (CPU) 12, host/PCI/cachebridge 20 and main memory 24.

The CPU 12 comprises one or more general purpose CPU cores 14 andoptionally one or more special purpose cores 16 (e.g., DSP core,floating point, etc.). The one or more general purpose cores executegeneral purpose opcodes while the special purpose cores executesfunctions specific to their purpose. The CPU 12 is coupled through theCPU local bus 18 to a host/PCI/cache bridge or chipset 20. A secondlevel (i.e. L2) cache memory (not shown) may be coupled to a cachecontroller in the chipset. For some processors, the external cache maycomprise an L1 or first level cache. The bridge or chipset 20 couples tomain memory 24 via memory bus 22. The main memory comprises dynamicrandom access memory (DRAM) or extended data out (EDO) memory, or othertypes of memory such as ROM, static RAM, flash, and non-volatile staticrandom access memory (NVSRAM), bubble memory, etc.

The computing device 11 also comprises various system components coupledto the CPU via system bus 26 (e.g., PCI). The host/PCI/cache bridge orchipset 20 interfaces to the system bus 26, such as peripheral componentinterconnect (PCI) bus. The system bus 26 may comprise any of severaltypes of well-known bus structures using any of a variety of busarchitectures. Example architectures include Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Associate (VESA) local busand Peripheral Component Interconnect (PCI) also known as Mezzanine bus.

Various components connected to the system bus include, but are notlimited to, non-volatile memory (e.g., disk based data storage) 28,video/graphics adapter 30 connected to display 32, user input interface(I/F) controller 31 connected to one or more input devices such mouse34, tablet 35, microphone 36, keyboard 38 and modem 40, networkinterface controller 42, peripheral interface controller 52 connected toone or more external peripherals such as printer 54 and speakers 56. Thenetwork interface controller 42 is coupled to one or more devices, suchas data storage 46, remote computer 48 running one or more remoteapplications 50, via a network 44 which may comprise the Internet cloud,a local area network (LAN), wide area network (WAN), storage areanetwork (SAN), etc. A small computer systems interface (SCSI) adapter(not shown) may also be coupled to the system bus. The SCSI adapter cancouple to various SCSI devices such as a CD-ROM drive, tape drive, etc.

The non-volatile memory 28 may include various removable/non-removable,volatile/nonvolatile computer storage media, such as hard disk drivesthat reads from or writes to non-removable, nonvolatile magnetic media,a magnetic disk drive that reads from or writes to a removable,nonvolatile magnetic disk, an optical disk drive that reads from orwrites to a removable, nonvolatile optical disk such as a CD ROM orother optical media. Other removable/non-removable, volatile/nonvolatilecomputer storage media that can be used in the exemplary operatingenvironment include, but are not limited to, magnetic tape cassettes,flash memory cards, digital versatile disks, digital video tape, solidstate RAM, solid state ROM, and the like.

A user may enter commands and information into the computer throughinput devices connected to the user input interface 31. Examples ofinput devices include a keyboard and pointing device, mouse, trackballor touch pad. Other input devices may include a microphone, joystick,game pad, satellite dish, scanner, etc.

The computer 11 may operate in a networked environment via connectionsto one or more remote computers, such as a remote computer 48. Theremote computer may comprise a personal computer (PC), server, router,network PC, peer device or other common network node, and typicallyincludes many or all of the elements described supra. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets and the Internet.

When used in a LAN networking environment, the computer 11 is connectedto the LAN 44 via network interface 42. When used in a WAN networkingenvironment, the computer 11 includes a modem 40 or other means forestablishing communications over the WAN, such as the Internet. Themodem 40, which may be internal or external, is connected to the systembus 26 via user input interface 31, or other appropriate mechanism.

The computing system environment, generally referenced 10, is an exampleof a suitable computing environment and is not intended to suggest anylimitation as to the scope of use or functionality of the invention.Neither should the computing environment be interpreted as having anydependency or requirement relating to any one or combination ofcomponents illustrated in the exemplary operating environment.

In one embodiment, the software adapted to implement the system andmethods of the present invention can also reside in the cloud. Cloudcomputing provides computation, software, data access and storageservices that do not require end-user knowledge of the physical locationand configuration of the system that delivers the services. Cloudcomputing encompasses any subscription-based or pay-per-use service andtypically involves provisioning of dynamically scalable and oftenvirtualized resources. Cloud computing providers deliver applicationsvia the internet, which can be accessed from a web browser, while thebusiness software and data are stored on servers at a remote location.

In another embodiment, software adapted to implement the system andmethods of the present invention is adapted to reside on a tangible,non-transitory computer readable medium. Computer readable media can beany available media that can be accessed by the computer and capable ofstoring for later reading by a computer a computer program implementingthe method of this invention. Computer readable media includes bothvolatile and nonvolatile media, removable and non-removable media. Byway of example, and not limitation, computer readable media may comprisecomputer storage media and communication media. Computer storage mediaincludes volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by a computer. Communication media typicallyembodies computer readable instructions, data structures, programmodules or other data such as a magnetic disk within a disk drive unit.The software adapted to implement the system and methods of the presentinvention may also reside, in whole or in part, in the static or dynamicmain memories or in firmware within the processor of the computer system(i.e. within microcontroller, microprocessor or microcomputer internalmemory).

Other digital computer system configurations can also be employed toimplement the system and methods of the present invention, and to theextent that a particular system configuration is capable of implementingthe system and methods of this invention, it is equivalent to therepresentative digital computer system of FIG. 1 and within the spiritand scope of this invention.

Once they are programmed to perform particular functions pursuant toinstructions from program software that implements the system andmethods of this invention, such digital computer systems in effectbecome special purpose computers particular to the method of thisinvention. The techniques necessary for this are well known to thoseskilled in the art of computer systems.

It is noted that computer programs implementing the system and methodsof this invention will commonly be distributed to users via Internetdownload or on a distribution medium such as floppy disk, CDROM, DVD,flash memory, portable hard disk drive, etc. From there, they will oftenbe copied to a hard disk or a similar intermediate storage medium. Whenthe programs are to be run, they will be loaded either from theirdistribution medium or their intermediate storage medium into theexecution memory of the computer, configuring the computer to act inaccordance with the method of this invention. All these operations arewell known to those skilled in the art of computer systems.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or by combinationsof special purpose hardware and computer instructions.

Tablet/Mobile Device Incorporating the Mechanism for Measuring theDistances Related to a Target Object

A high-level block diagram illustrating an example tablet/mobile deviceincorporating the distance measuring mechanism of the present inventionis shown in FIG. 2. The mobile device is preferably a two-waycommunication device having voice and/or data communicationcapabilities. In addition, the device optionally has the capability tocommunicate with other computer systems via the Internet. Note that themobile device may comprise any suitable wired or wireless device such asmultimedia player, mobile communication device, digital still or videocamera, cellular phone, smartphone, iPhone, PDA, PNA, Bluetooth device,tablet computing device such as the iPad or other iOS device, Androiddevice, Surface, Nexus, Google Glass, etc. For illustration purposesonly, the device is shown as a mobile device, such as a cellular basedtelephone, smartphone or superphone. Note that this example is notintended to limit the scope of the mechanism as the invention can beimplemented in a wide variety of communication devices. It is furtherappreciated the mobile device shown is intentionally simplified toillustrate only certain components, as the mobile device may compriseother components and subsystems beyond those shown.

The mobile device, generally referenced 60, comprises one or moreprocessors 62 which may comprise a baseband processor, CPU,microprocessor, DSP, etc., optionally having both analog and digitalportions. The mobile device may comprise a plurality of cellular radios102 and associated antennas 104. Radios for the basic cellular link andany number of other wireless standards and Radio Access Technologies(RATs) may be included. Examples include, but are not limited to, CodeDivision Multiple Access (CDMA), Personal Communication Services (PCS),Global System for Mobile Communication (GSM)/GPRS/EDGE 3G; WCDMA; WiMAXfor providing WiMAX wireless connectivity when within the range of aWiMAX wireless network; Bluetooth for providing Bluetooth wirelessconnectivity when within the range of a Bluetooth wireless network; WLANfor providing wireless connectivity when in a hot spot or within therange of an ad hoc, infrastructure or mesh based wireless LAN (WLAN)network; near field communications; UWB; GPS receiver for receiving GPSradio signals transmitted from one or more orbiting GPS satellites, FMtransceiver provides the user the ability to listen to FM broadcasts aswell as the ability to transmit audio over an unused FM station at lowpower, such as for playback over a car or home stereo system having anFM receiver, digital broadcast television, etc.

The mobile device may also comprise internal volatile storage 64 (e.g.,RAM) and persistent storage 68 (e.g., ROM) and flash memory 66.Persistent storage 68 also stores applications executable byprocessor(s) 62 including the related data files used by thoseapplications to allow device 60 to perform its intended functions.Several optional user-interface devices include trackball/thumbwheelwhich may comprise a depressible thumbwheel/trackball that is used fornavigation, selection of menu choices and confirmation of action,keypad/keyboard such as arranged in QWERTY fashion for enteringalphanumeric data and a numeric keypad for entering dialing digits andfor other controls and inputs (the keyboard may also contain symbol,function and command keys such as a phone send/end key, a menu key andan escape key), headset 88, earpiece 86 and/or speaker 84, microphone(s)and associated audio codec 82 or other multimedia codecs, vibrator foralerting a user, one or more cameras and related circuitry 110, 112,display(s) 122 and associated display controller 106 and touchscreencontrol 108. Serial ports include a micro USB port 76 and related USBPHY 74 and micro SD port 78. Other interface connections may includeSPI, SDIO, PCI, USB, etc. for providing a serial link to a user's PC orother device. SIM/RUIM card 80 provides the interface to a user's SIM orRUIM card for storing user data such as address book entries, useridentification, etc.

Portable power is provided by the battery 72 coupled to power managementcircuitry 70. External power is provided via USB power or an AC/DCadapter connected to the power management circuitry that is operative tomanage the charging and discharging of the battery. In addition to abattery and AC/DC external power source, additional optional powersources each with its own power limitations, include: a speaker phone,DC/DC power source, and any bus powered power source (e.g., USB devicein bus powered mode).

Operating system software executed by the processor 62 is preferablystored in persistent storage (i.e. ROM 68), or flash memory 66, but maybe stored in other types of memory devices. In addition, systemsoftware, specific device applications, or parts thereof, may betemporarily loaded into volatile storage 64, such as random accessmemory (RAM). Communications signals received by the mobile device mayalso be stored in the RAM.

The processor 62, in addition to its operating system functions, enablesexecution of software applications on the device 60. A predetermined setof applications that control basic device operations, such as data andvoice communications, may be installed during manufacture. Additionalapplications (or apps) may be downloaded from the Internet and installedin memory for execution on the processor. Alternatively, software may bedownloaded via any other suitable protocol, such as SDIO, USB, networkserver, etc.

Other components of the mobile device include an accelerometer 114 fordetecting motion and orientation of the device, gyroscope 115 formeasuring or maintaining orientation, magnetometer 116 for detecting theearth's magnetic field, FM radio 118 and antenna 120, Bluetooth radio 98and antenna 100, Wi-Fi radio 94 including antenna 96 and GPS 90 andantenna 92.

In accordance with the invention, the mobile device 60 is adapted toimplement the distance measurement and image processing mechanism ashardware, software or as a combination of hardware and software. In oneembodiment, implemented as a software task, the program code operativeto implement the distance measurement and image processing mechanism isexecuted as one or more tasks running on processor 62 and either (1)stored in one or more memories 64, 66, 68 or (2) stored in local memorywithin the processor 62 itself.

Measurement of Distances Related to a Target Object and Related ImageProcessing System

A block diagram illustrating an example room in which an end userobtains a digital image of sample window is shown in FIG. 3. Thedistance measurement mechanism enables the automatic measurement ofdimensions for a window part from a digital image of the window thatincludes a reference object in substantially the same plane as theconstraint dimension associated with the window. The end user 131 takesa photograph, using a digital image acquisition device such as digitalcamera 133, which includes a window 130 and the reference object 132.Knowledge of the reference object dimensions is used to calculate anydimensions needed for fabrication of one or more parts for fenestration.The image processing calculations may be performed on the digital imageacquisition device such as a smartphone or tablet with built-in camera,on an end user's PC, an external website or any other computing deviceafter the image is uploaded to it.

Also shown in FIG. 3 are one or more ancillary objects placed on thewindow. Ancillary objects 136 and 138 are typically placed on the windowsash or frame and function to aid in demarcating points for measurement.Another ancillary object 134 is placed at the top of the window andfunction to aid in determining the location of the top edges of thewindow when a window treatment (e.g., blind) is installed on the window.

A block diagram illustrating an example network showing the data flowbetween fabricator, designer, service provider and end user is shown inFIG. 4. The network, generally referenced 140, comprises an end user162, PC or other computing device 150 connected to the Internet or otherwide area network 148, fabricator 142, designer 144 and service provider146. End users may also be connected, for example, via smartphone 158running an appropriate application (i.e. app) or a tablet device 160running an appropriate app. Both the smartphone and tablet are connectedto the internet via cellular base stations 156 and the cellular network154. Note that the tablet and smartphone may be connected to theInternet through Wi-Fi to an access point that is connected to theInternet.

End users communicate with the fabricator, designer and service providervia the Internet and connect via any number of devices such as a tablet(e.g., iPad device, Android device, Surface, Nexus, etc.) connected viaWi-Fi or through a cellular connection, desktop/laptop (via wired orwireless connection) computer, mobile device such as a smartphone orcellular enabled wireless tablet both in communication with thefabricator, designer and service provider via cellular network (e.g.,3G, 4G, etc.) including base stations.

The fenestration measurement and image processing mechanism provides thecapability of accurately measuring and determining the dimensions of oneor more parts from a digital image. The system is intended for use onany computer system such as desktop computers, laptop computers,notebook computers, netbook computers, ultrabook computers, wirelessmobile devices, mobile phones, tablets, iOS devices, Android devices,Firefox OS devices, etc. It is however, especially applicable for use ontablets and mobile devices such as the Apple iPad, Android based tabletssuch as Google Nexus, Microsoft Windows tablets such as the Surface andother tablet formats or smartphones such as the Apple iPhone, Androidbased smartphones or Windows based smartphones.

Throughout this document the term “website” is used to refer to auser-accessible network site that implements the basic World Wide Webstandards for the coding and transmission of hypertext documents. Thesestandards currently include HTML (the hypertext markup language) andHTTP (the hypertext transfer protocol). Note that the term “site” is notintended to imply a single geographic location as a website or othernetwork site can, for example, include multiple geographicallydistributed computer systems that are appropriately linked together.

It is to be understood that elements not specifically shown or describedherein may take various forms well known to those skilled in the art.Figures provided herein are given to show overall function, operation,and relationships and are not drawn with the intention of showingcomponents or elements to scale. It is also to be understood that whilethe figures and descriptions provided relate to windows andmodifications to windows, the method of the present invention may beused in the design, fabrication or specification of any objects meant towork with, within or to replace a target object having one dimensionthat is substantially smaller than the other two dimensions or having asubstantially planar face.

Various terms are used in the art to describe aspects of fenestrationand windows in particular. In describing the present invention, the term“window” may refer to a single frame, one or more frames within acomplex or an entire complex frame. A “complex” frame refers to multiplewindowpanes within the same frame. In describing the present invention,the terms “interior” and “exterior” are used to describe the indoor sideand outdoor side, respectively, relative to a perimeter wall in whichthe fenestration resides. “Inward” and “outward” facing refers to framesurfaces perpendicular to the perimeter wall plane facing toward or awayfrom, respectively, the center of the fenestration.

The term “overlay” is defined as designed to cover an interior orexterior side of a windowpane using as support surfaces such as sash,interior facing trim casing or wall surfaces and includes surfaces thatmay reside between a screen and windowpane of, for example, casement orawning windows. The term “in-frame” is defined as designed to cover aninterior or exterior side of a windowpane using for support surfaces of,for example, jambs or jamb liners, sash channels, stops or inward facingsurfaces of trim casing.

The terms “supplemental part” or “supplemental element” are defined asan article that is designed for use with a target object. Non-limitingexamples of supplemental parts may include window treatments, films,overlays or inserts that enhance the aesthetics, light control, or heattransfer of windows, or may also include paint, wallpaper, cabinets,shelving, frames or furniture.

The term “sealing interface” is used to describe a visible portion of awindow, that may be reversibly opened and closed, at which reversiblesubstantial sealing occurs, when viewed from either the interior orexterior.

The terms “automated”, “semi-automated” and “manual” are used todescribe different degrees of human intervention in a process by anend-user, professional or service provider. “Manual” refers to a processperformed entirely by a human; “automated” refers to a process performedentirely by computational or other electronic devices; and“semi-automated” refers to a process involving computational or otherelectronic devices with human intervention at a point in the process.

Note that various people or entities may perform different aspects ofthe present invention. An “end-user” refers to a person or entity ortheir designee, that specifies, orders, installs or uses thesupplemental parts of the present invention and may perform digitalimage capture, supply metadata and/or confirmation of design steps ofthe process of the present invention. A “service provider” refers to aperson or entity performing a service that is part of the method of thepresent invention such as reviewing and accepting or confirming ordersfrom an end-user, providing image processing capability, designing (as a“designer”), fabricating (as a “fabricator”) or installing (as an“installer”) parts, or providing support for installation of such parts.In the present invention, a “service provider” may provide to an “enduser” instructions or directions, including, but not limited to, forobjects to be used, for printing on and/or placement of objects in ascene, for capturing of the scene in a digital image, and foridentifying locations of objects in a digital image of a captured scene,such that the instructions or directions may be carried out by the “enduser” at or near the location where the digital image is captured. Otheraspects of the present invention relate to dimensions of objects to bemeasured or imaged. A “target object” of the present invention refers toan object having a constraint dimension that is measured by one or moremethods of the present invention. A “reference object” is an object thatis used to estimate a pixel scale factor (or calibration factor) for thecaptured image. In describing the present invention, “constraintdimension” refers to a measured portion or a multiple of a measuredportion of a target object to which a designed part is to conform and a“constraint pixel dimension” refers to the length of a constraintdimension measured in pixels. Similarly, “reference dimension” refers toa reference object dimension whose bounds are detectable in a captureddigital image and a “reference pixel dimension” refers to a referencedimension measured in pixels. A target object may contain a “symmetryelement” which in the present invention refers to an aspect of thetarget object that in standard practice resides at a position within thetarget object such that the symmetry element divides a constraintdimension in an integer number of equal parts.

An “ancillary object” of the present invention refers to an object thatis used to aid in finding the location of an item to be measured in thecaptured image, including for example, the target object to be measuredor an edge or corner thereof. The invention provides for three types ofancillary objects. The first is a pane ancillary object to aid indetermining the location of corners or edges of a portion of the windowsuch as a sash or muntin. Pane ancillary objects may be placed, forexample, in diagonal corners of the window to aid in determining thewindow dimensions. The second is a sealing interface ancillary objectfor aiding in determining the location of a sealing interface portion ofthe window. The third is a non-transparent target object ancillaryobject used for identifying locations on a target object other than awindow, for example. Non-transparent target object ancillary objects maycomprise (a) frame ancillary objects for aiding in identifying windowframe edges for measurements, such as inward facing frame or jamb edgesor outward facing frame casing edges; or (b) wall ancillary objects foraiding in identifying wall edges or portions of a wall for measurement.

A “contrast object” functions to provide contrast between existingelements in a captured image. For example, contrast objects are used sothat the edges of the reference object can be found in the event therewould otherwise be little contrast between the reference object and itsbackground.

An “adhering object” functions to keep the reference object applied to aportion of the scene in the captured image, e.g., a wall, window frame,window, etc.

Embodiments of the present invention contemplate improved method andapparatus for decreasing heat transport through fenestration in whichthe method of obtaining measurements for custom manufacturing of theinsulation and its support is done through photogrammetry using digitalimages and digital image processing. Other embodiments of the presentinvention contemplate improved methods and apparatus for supporting,storing and re-using the insulating materials. While the descriptionprimarily discusses embodiments related to windows as target objects,other embodiments may include other planar target objects such as awall, ceiling, floor, furniture or portions thereof, artistic painting,poster, photograph, appliance, or any other object where it is desiredto estimate a constraint distance or dimension.

Target Object Measurement Digital Image Processing

One aspect of supplemental window elements that is critical is theattainment of accurate measurement fenestration attributes for propermatching of the supplemental window element to the fenestration.Necessary measurements may include physical dimensions such as width,height and depth as well as color. Such measurements, however, can betime consuming and difficult to achieve for those users not accustomedto such work or if the installation site is difficult to access.Depending on the approach, a significant amount of material may bewasted, either from mismatch of delivered product and the area to becovered or from errors made by end users having insufficient fabricationand installation experience. Further, the presence of objects such asfurniture or existing window treatments may complicate attainment ofrequisite measurements. In addition, depending on the type of window,frame and window treatment, supplemental windows may be difficult orimpossible to properly install for optimal thermal and radiativeinsulation.

While prime windows (e.g., single and multiple pane windows generallyusable on a stand-alone basis in fixed buildings, mobile homes, traveltrailers and other habitations) are sufficient for structural integrityand habitation security, they are often found to be an insufficientthermal and radiation barrier. To conserve the energy necessary forheating and/or cooling a building supplemental windows are employed inaddition to the prime windows. Such supplemental windows have includedexterior and interior “storm” windows mounted over the prime windowswith a “dead air” space therebetween.

Supplemental windows are structurally and functionally distinct fromprime windows. Supplemental windows are primarily intended to protectthe prime window and reduce thermal losses therethrough. In manyinstances, supplemental windows are intended to be installed by thebuilding owner and/or relatively inexperienced workers. As a result,supplemental windows are preferably lightweight, uncomplicated andinexpensive. To avoid detracting from the appearance of either thebuilding in general or the prime window itself and to fit within oftentight pre-existing spatial constraints, supplemental windows have tendedto have minimal framework, the visible bulk of the window assembly beingthe window panes. Also, “weep holes” or passageways from the environmentto the dead air space are usually provided to avoid condensation buildup between the exterior storm window and the prime window. Thus, anoptimal thermal barrier between the windows is not achieved.

Interior storm windows can be installed regardless of building heightand legal restrictions on exterior building appearance, but suffer otherdisadvantages. Such windows are generally mounted within the windowopening or on the interior building wall outside of the window opening.In such cases these windows are preferably constructed with frames fromplastic material, such as vinyl, to reduce thermal conductivity, weight,and expense. These materials, however, have been found to sag and warpin response to the weight and thermal stresses particularly in largewindows subject to extended periods of direct sunlight. This sagging isdestructive of the structural and air seal integrity of the window unitand can increase the difficulty of raising or lowering the window panes.Further, in tall windows vinyl material has been found to lacksufficient rigidity to maintain close air seals between the sides of thewindow pane and the receiving channels. Moreover, in those instanceswhere such windows are installed within the window opening, customsizing and installation are typically needed for each window opening,especially when retrofitting such storm windows to older buildings.

In one embodiment, a customer who wishes to have custom windows orsupplemental materials must provide the vendor with window dimensions.Alternatively, an estimator/installer obtains the dimensions. Thesedimensions are manually input by a skilled operator into a computeraided design device (commonly referred to as a CAD) that creates anelectronic image which in turn is input to a plotter/cutter. Theplotter/cutter generates the sheet of film cut to the customspecifications. The film is then applied to the window by the customeror installer. Alternatively, the customer or estimator/installer mayinput the dimensions into an input device and directly receive the cutfilm without utilizing the services of a skilled operator through aservice such as www.computercut.com. Such a service provides the cutfilm order created at a location remote from the source of the film andthen sent (by mail, courier, etc.) to the requestor at the remotelocation.

Note that using other methods other window related custom products suchas window treatments or coverings are efficiently delivered. Windowcoverings are sold in standard sizes by department stores, discountstores and home centers. They are also sold by custom fabricators whocome to the home or office, measure the windows and make blinds to fit.Some retailers sell custom blinds based upon measurements provided bythe customer. These retailers keep a limited inventory of stock blindsin standard sizes and popular colors. If the customer does not want ablind from the retailer's current inventory, the retailer may customorder the blind from the manufacturer using the customer's measurements.

Stock blinds have a standard width and length and come in a limitednumber of colors and materials. In a stock blind, lift cords and tiltcontrols, if any, are in the same location on every blind. In a customblind, the blind is made to have a length and width that corresponds tothe size of the window opening. The customer specifies whether the liftcords and tilt control are to be on the left side or right side of theblind to avoid nearby secondary objects. The customer can often obtain acustom blind in colors not available in stock blinds. Other options maybe available to the buyer of a custom blind that are not available in astandard or stock blind.

The alternative window coverings (“AWC”) industry provides soft and hardwindow treatments to customers desiring window coverings other thanconventional draperies. Hard window treatments include faux wood andwood horizontal blinds, vinyl and metal horizontal blinds, verticalblinds and interior shutters. Soft window treatments include cellularshades, pleated shades, roller shades, soft shades, vertical blinds andsoft window shadings. AWC products are offered to customers through avariety of retail channels, including home product centers, independentretailers, discount department stores, retail fabricators, departmentstores, catalogs, internet, home builders and interior designers anddecorators. Typically, custom-made products are manufactured by awholesale fabricator or a retail fabricator and then are sold eitherdirectly to customers or to a retail source that, in turn, sells thecompleted product to the customer.

A customer desiring a custom-made window covering typically places anorder with a retail source, specifying the features of the finishedproduct desired. Such features can include information about the size ofthe window, the style, the desired color and various additional optionsincluding the type of hardware to be included for mounting andcontrolling the window covering after installation. The retail sourcepasses the order along to the fabricator. Upon receiving the order, thefabricator cuts the pre-colored bulk material into the size specified bythe customer and adds the desired hardware to produce the custom windowcovering. The completed product is then sold directly to the customerand/or shipped to the retail source.

This fabrication technique has disadvantages for the fabricator. Notabledrawbacks include wasted inventory due to the generation of scrapmaterial in the manufacturing process and obsolescence of inventory dueto changes in manufacturer color lines. The cost of this wastedinventory is typically absorbed by the fabricator and is typicallypassed along to the end user or customer.

A diagram illustrating a sample window and reference dimensions areshown in FIG. 5. The window, generally referenced 170, comprises thewall 172, frame casing 174, top and bottom sash window 176 with muntins,is shown with a reference object 178 on the lower sash and threeancillary objects, one ancillary object 179 is on the upper sash whiletwo other ancillary objects 175, 177 are placed at sealing interfaces atthe left side and bottom, respectively, of the lower sash.

A diagram illustrating the volume of space an image acquisition devicemust be in when acquiring the digital image of the window is shown inFIG. 6. Since the proficiency of end-users capturing the digital imagesmay be highly variable, there are a number of aspects of the imagecapture that are preferred in order to keep measurement error to aminimum. It is preferable for the camera (image acquisition device) 454to be substantially within the orthogonal projection 452 of the targetobject 450 (e.g., window) toward the image acquisition device, in thiscase substantially within the cuboid volume extending from the windowopening into the room in which the window exists, so that the imagingplane is nearly parallel to the plane in which the target objectwindow/fenestration resides. It is also most preferred that the imageacquisition device be positioned at or very near the center of theorthogonal projection of the target object toward the image acquisitiondevice.

It has been found that images captured outward from the constraintprojection, in this case the window trim casing, can lead to distortionsthat are difficult to correct without leaving distortion in thereference and/or constraint dimensions or may render a constraint edgehidden in the captured image. To aid with this positioning for imagecapture, it can be helpful to capture the image with minimal or nobacklighting so as to make reflection of the person capturing the imagereadily visible to this person when within the projection of the windowopening. Further, it is more preferred that the camera reside close tothe projection of the window/fenestration center. The capture of imageswith the camera near the fenestration center also aids in embodiments ofthe present invention where vanishing point methods are employed tocalculate supplemental part dimensions. When employing vanishing pointmethods, lines perpendicular to the plane of the fenestration such asthose associated with the sill, stool, check rail top edges of the lowersash of a vertically operable sash, and inward facing stop edges can beused. Additionally, for reasons discussed below, it is preferred to usean image capture device that allows for minimization of camera motionduring exposure. The image capture device may comprise a still camera,video camera, sequence of still images taken in rapid fire fashion,smartphone camera, etc.

Since windows are generally transparent and rectangular in shape theyoffer the opportunity for further automation of distance measurement. Bycapturing the digital image under conditions of either predominantlyfront lighting or predominantly back lighting of the window, highcontrast portions of the image are easily obtained and identified.Front-lit images with minimal or low levels of back lighting (forexample, taken at night) can be advantageous for choosing customsupplemental part color with respect to the wall, frame and or existingwindow treatment, easier identification of details in frame molding thatmay affect mounting, and minimizing shadows that could adversely impactchoice of measurement points if minimal image processing is used. Inaddition, having a dark background eliminates the potential forirrelevant rectangular shapes to be present in captured digital imagesthus simplifying the process of identifying relevant features, such as areference object, a frame or sash element or muntin. Thus, capturing theimage at nighttime with room lighting or with flash illumination, thetransparent portion of the window will appear very dark with respect toa light colored window sash. Such lighting conditions also allow theperson capturing the image to adjust the camera position within theframe projection by observing the location of the camera reflection.Alternatively, a capture device with capability of capturing bothvisible and infrared images may be used. In such a case, capturing theimages at a time when there is a significant temperature differentialbetween the exterior and interior sides of the window may allow regionsof interest, such as pane edge locations or sealing interfaces to befound in the infrared image. Using the spatial relationship of thevisible and infrared images, the regions of interest may be found andused in the image processing of the visible image.

An alternative method of the present invention provides referencedimension measurement using a reference object, optionally havinganother use when not used in the present invention, or may be a standardsize reference object. Prior to capturing the digital image, the enduser may place a standard sized object on the window frame, sill, stool,sash, windowpane, next to the window or within the window frame beingphotographed, as shown in FIG. 3. Standard sized objects should have aneasily identified linear dimension that is viewable in the image. Morethan one standard sized object may be used in an image. Non-limitingexamples of such standard sized objects include an open tape measure, aruler or meter stick, a piece of printing paper or lined paper havingknown standard dimensions, e.g., letter, legal, A4, A5, etc., a CD jewelcase, currency, credit or debit card or government issued documents suchas a driver's license or passport. When using an object similar in coloror that does not provide sufficient contrast with its surroundingelements it is preferred to have a high contrast border at theperipheral edges or high contrast lines that terminate at the its edge.The reference object may also be a thin electronic display device suchas a tablet or laptop computer display or a cell phone display for whichthe make and model is known and conveyed to the service provider asmetadata. Such a display may be altered to provide high contrast and/orcolor distinction from the surrounding primary and secondary objects toaid in finding and dimensioning such a reference object. Alternatively,a standard object or figure provided by the service provider may beused, printed or displayed electronically whereby the service providerpredetermines the dimensions of the standard object or printed figure.When a standard object is provided by the service provider, suchstandard object is preferably planar and rigid or semi-rigid and mayoptionally have printed on it a standard figure.

In one embodiment, a standard object or figure may have an uncommoncolor defining the standard length so that the end user may capture adigital image of the standard object or figure that will subsequently beused as the reference object in the present invention. Using the samecapture device and colored standard object and providing their identityto the service provider in the present invention can then aid inautomated locating of the reference object in one or more digital imagesused in the present invention. Additionally, the end user may create areference object by measuring a non-standard sized object's dimensionsand supplying the reference dimensions to the service provider asmetadata. Similarly, color information may be calibrated by providingthe end user with a standard color sample that can be used to calibratecolors in the image. Examples of objects predetermined by the serviceprovider include pre-printed paper, plastic sheet, picture frame, clipboard, cork board or bulletin board sent to or otherwise obtained orpurchased by the user and digital files that may be printed by the usernear the point of use. When the user prints digital files provided bythe service provider, the digital file may be printed on a standard sizesheet of paper such that the sheet of paper acts as the reference objectand the printed file provides means for identifying the sheet of paper.In such cases, digital files preferably comprise at least one fiducialpattern, such as a checkerboard, dot, or hourglass pattern, or a barcode or QR code.

Fiducial patterns may perform one or more functions in the presentinvention, such as enabling automated object or feature finding,automated orientation of objects and the image, containing user and/ororder information, or enabling relative sizing of different parallelplanes. For encoding user and/or order information the fiducial maycomprise a code such as a bar code or QR code, the information of whichmay optionally be encrypted by the service provider, printed largeenough for resolving in an image sent from the end user to the serviceprovider. For the relative sizing function, it is particularly helpfulto print fiducial patterns having the same pixel dimensions andorientation in the digital file with the same printer settings on thesame printer. Such printed digital file preferably contains orientationinformation such that, when mounted properly by the user, leads toorientation information in the captured image. In addition, the fiducialpattern is printed large enough for the pattern to be found andinterpreted by computer vision software.

In one embodiment, checkerboard and hourglass patterns (shown in FIG. 5)of at least about one inch squares have been found to be useful. In thecase of QR codes, so that reading of the code at or above about 10pixels per inch in the image, the pattern should be at least about twoinches by two inches in size for a 13×13 pattern, and preferably aboutsix inches by six inches for a 29×29 pattern. Preferably, referenceobjects should be rigid or can be made rigid during the image capture.For example, if a piece of printing paper is used, at least two,preferably three, adjacent corners should be taped to a flat surfacewith the entire edge between the adjacent corners in contact with theflat surface. Alternatively, standard sized printing paper may beattached to the window pane, muntins or sash using small amounts ofliquid, gel, paste, crème or grease applied to the non-printed side ofthe printing paper, preferably near the edges of the paper so as to keepthe paper as flat as possible. Liquids useful for this purpose do notpermanently adhere and are easily cleaned off of the window pane, muntinor sash to which they contact and may otherwise be easily found by theuser in their home or workplace. Such liquids include water, althoughpreferred liquids have higher viscosity than water such as dishwashingliquid, liquid soap, hair shampoo or conditioner, hand lotions, oils(for example, vegetable based oils). Preferred gels, pastes, crémes andgreases include petroleum jelly, lip balm or gloss, toothpaste orointment.

When attaching printing paper to muntins that protrude toward the roominterior from the window pane, it is preferred to have at least fourattachment points. To minimize curling of the printing paper, such astypical 8.5×11 inch multipurpose 24 lb. white paper, after the fiducialpattern has been printed, printing methods that minimize or eliminatewater contact with the printing paper, such as toner based printing, arepreferred. Use of standard size heavier weight paper or card stock isalso beneficial for minimizing curling. When printing with methods thatcan lead to curling of the paper, such as inkjet printing, it can bebeneficial to decrease the amount of printed material used to print thefiducial pattern, while maintaining detectability by the imageprocessing algorithm. This may be accomplished by employing, forexample, halftoning to adjust the amount of material deposited on thepaper to achieve the desired balance between sheet flatness anddetectability of the printed pattern. While bi-tonal black-and-whitepatterns have been successfully used in the present invention, halftonedpatterns may be used in the present invention. Gray levels in the 50% to75% range have been found to provide improved sheet flatness ofreference and ancillary objects while providing sufficient density foralgorithmic detection of fiducial patterns under many lightingconditions. Higher gray levels, such as about 80% to 100% (black)improve detectability by the methods of the present invention,particularly for the ancillary objects that may be partially hidden orcovered by a shadow.

When using a reference object, it is preferred to place the plane of thereference dimensions of the reference object as close as possible andparallel to the plane of the measured constraint. Therefore, referenceobjects that are thin in the dimension parallel to the constraint planeare preferred. If the reference object is placed outside thefenestration, for example on the wall immediately next to thefenestration, as described below, the fiducial pattern on the referenceobject may be used to aid in locating regions of interest and thereference object edges, particularly if there is low contrast betweenthe reference object and the wall. In addition, entropy methods maybeneficially be used to aid in differentiating the reference object,ancillary objects, adhering objects or contrast objects from theadjacent features such as the wall, or when there is low contrast withthe window features or windowpane background, adjacent to the referenceobject, ancillary object, adhering object or contrast object in theimage in the region of interest, so that reference object edges may befound more easily.

If the reference dimensions are not placed in the same plane as theconstraint dimensions, size correction may be performed to account forthe perspective error induced by such placement. One method ofperforming such correction is to print the fiducial patterns using thesame printer with the same print settings and having fiducials that aredigitally the same size as the pattern printed on the standard sizeprinting paper. The fiducial on the standard size printing paper may becalibrated to the known dimensions of the paper. Placing a second(ancillary) fiducial in a second plane parallel to the window allowsdimensions in the second plane to be correctly measured. Preferably,such reference objects are placed near window dimensions of similarlength to be determined.

The captured and processed images should have a resolution of greaterthan one megapixel, preferably greater than two megapixels, morepreferably greater than three megapixels and most preferably greaterthan four megapixels. At the same time, to facilitate edge and corneridentification and decreased camera motion errors, reference pixeldimensions must be of sufficient length relative to the image pixeldimensions. Through extensive experimentation capturing digital imagesusing imaging devices of different resolution, reference objects ofdifferent dimensions, and different image plane to fenestration planedistances, it has been found that the reference object and itsdimensions must be carefully chosen and placed so that symmetricalelements and constraint elements may be readily observed.

If a target object window already has an associated window treatmentthat will be used with the custom supplemental parts, the image ispreferably captured with the treatment opened allowing constraintsurfaces and lines to be visible. If the open treatment still covers aportion of the window or frame, additional images of the coveredportions may be captured to obtain constraint surfaces or lines hiddenin other image views. Any additional image should also contain areference object so that accurate calculations may be obtained.

In some cases it may be desirable to capture only a single image but theimage may have omitted a portion of a relevant constraint, such as acorner or edge. In other cases, a window treatment may be in a fixedposition covering at least one of the constraint surfaces or lines. Insuch cases, symmetry within the window and/or framing or digitalextension of the observable constraints may be used to calculate adimension for which a portion of one constraint is not visible in theimage. Symmetry elements such as check rails or muntins may be used toestimate the location of completely hidden constraints. Alternatively,one or more ancillary objects may be created to provide a means fordetermining the location of a hidden constraint. For example, a fiducialpattern provided by the service provider may be printed on a piece ofstandard sized printing paper, in which the ancillary object comprises afiducial pattern and at least one of the intact standard dimensions ofthe printing paper. For example, one edge of the 11 inch dimension of an8.5×11 inch piece of printing paper upon which a fiducial is printed maybe aligned with the edge of a window pane that is partially obstructedby, for example, a window treatment. When so aligned, the edge oppositethe aligned edge may be visible while the aligned edge is obstructed.The fiducial pattern may be used to locate this ancillary object whosestandard length is known. The intact dimension from the visible edge tothe aligned edge is a standard dimension which, along with knowledge ofthe pixels per inch for the plane in which this ancillary objectresides, allows the hidden constraint location to be determined.

In cases where a window treatment is moveable and covers differentportions of constraint surfaces or lines when in different positions, itcan be beneficial to capture more than one image of the same window suchthat different treatment positions are captured. The end user may selectand adjust treatment positions to be captured such that the imagesprovide complementary views of constraints. Software programs may beemployed to merge two or more images creating a single image offering aclear view of the all desired constraint surfaces or lines in a singleimage. For example, vertical or horizontal blinds may allow imagecapture with partial view of a constraint rectangle when raised orpulled to the sides of a window. One constraint surface, however, may bepartially or entirely hidden with the blind in such a position. Tocomplement this image, the blind may be in its fully closed positionwith the blinds rotated to allow imaging of a constraint surface that ishidden in the first image. This single image, having the non-stationarytreatment portions removed, may then be used as the basis for furtherimage processing described below.

A preferred embodiment is now described. In this description, severalterms are used with the following definitions. A triage resolution imageis an image with a resolution suitable for scene content analysis; forexample, an image with a width of 960 pixels and height of 640 pixelswill provide a resolution of 8 pixels per inch for an image fillingobject that is 120 inches wide and 80 inches tall in a plane parallel tothe image plane and will provide higher resolution for closer objects. Ameasurement resolution image is an image with a resolution suitable fortarget object measurement; for example, an image with resolution of 12pixels per inch at the target object distance.

A measureable image is an image in which rectangular objects in theworld appear rectangular in an image; for example, an image of arectangular object directly in front of a camera where the axis of thelens is perpendicular to the rectangular object. A projectivetransformation is a mapping from points in one coordinate system topoints in another coordinate system which preserves collinearity; forexample, a pinhole camera photograph of a planar object on the imageplane of the camera. A projectivity matrix is a matrix which is used toencapsulate the calculations needed to perform a projectivetransformation; for example a 3×3 matrix that is used to convertlocations (u, v) in one plane to locations (x, y) in another plane byadding a third coordinate of 1 to a source point (u,v,l) and computingthree coordinates (hx, hy, h) by matrix-vector multiplication from whichthe coordinates of the destination point (x, y) are computed bydivision.

Image coordinates are the location of a point of interest in a digitalimage, typically given in pixel offset units from a defined origin pointsuch as the upper left corner of the image; for example, if an image has960 columns and 720 rows of pixels, the offset of the image center fromthe upper left corner of the image is 479.5 columns and 359.5 rows.Camera coordinates are an extension of image coordinates to threedimensions by adding distance from the image plane as a thirdcoordinate. Plane coordinates are the location of a point of interest ona planar object in physical units from a defined origin point. Forexample, a planar object being photographed may have its coordinatesdefined as offsets from the point of intersection of the camera axiswith the plane with distances measured in centimeters or inches. Worldcoordinates are an extension of plane coordinates to the location of athree dimensional point of interest relative to a plane wherein thefirst two coordinates are the coordinates of the point on the planeclosest to the point of interest and distance to the plane is the thirdcoordinate.

For example, if a plane coordinate system has its axes defined using twoadjacent sides of a rectangular object such as a window pane, apane-relative world coordinate system could be defined in a way thatpositive third coordinates refer to points outside the window andnegative refer to points inside. Translation changes in coordinates dueto shifting a coordinate system origin point without changing theorientation or units of the coordinate axes. Scaling changes incoordinates due to a change in units on one or more coordinate axeswithout shifting or reorienting the coordinate axes; equal scaling ofall dimensions is called an isotropic scaling; an isotropic scalingpreserves the aspect ratio of rectangular objects. Rotation changes incoordinates due to changing the orientation of a set of coordinate axeswithout shifting or changing units of the coordinate axes.

A capture mapping is a projective transform that maps points in a planeor world coordinate system onto points in an image. An inverse mappingis a projective transform which maps points in an image plane or cameracoordinate system onto points in a plane or world coordinate system.

Flow diagrams illustrating an example overall workflow between the enduser and service provider are shown in FIGS. 7A and 7B. Followinginstructions provided by the service provider after accessing theservice provide website or mobile app (step 180), the end user printsdigital files of fiducial patterns (provided by the service provider(step 204)) on standard printing paper (step 182). The end user suppliesmetadata such as the particular type of window and the product desired(step 184). For example, the end user may answer questions to supplymetadata or the metadata may be the result of using a trial and errordigital application such as “The Window Shopper”, available fromwww.blinds.com, that aids the end user in choosing a window relatedproduct with the desired mounting location. Such mounting location maythen be input to the instructions provided and used in the presentinvention.

The user places on the window pane or protruding muntins a referenceobject of standard printing paper, such as 8.5×11 inch, on which afiducial pattern is printed (step 186). In addition, the end useroptionally places ancillary objects, of standard or non-standard size onwhich fiducial patterns are printed, on window components as instructedby the service provider, consistent with metadata provided by the enduser. The ancillary object fiducial patterns are distinct from thereference object fiducial pattern. Preferably, multiple ancillaryfiducial objects may be printed on a single standard size sheet ofprinting paper and separated by cutting or tearing along linespre-determined by the service provider.

After constructing the scene of the window by moving obstructing objectsand placing reference and ancillary objects in the scene, the end usercaptures the digital image from a point within the projection of thewindow as described above in connection with FIG. 6 (step 188).

The captured image is resized to a triage resolution, for example sothat the smaller dimension of the image has 720 pixels or 720 P, andeither sent to a remote server for automated image processing orautomated image processing may take place on the capture device (step190). While the downsized image is being processed, transmission of ahigh resolution version of the image, which may be the full resolutionof the imager, to the service provider server may commence (step 196)and, if transmission is completed the high resolution image may bestored (step 198). The service provider analyses the triage resolutionimage and provides feedback to the user and determines whether a highresolution image is needed (step 206). If recapture is required (step192), the method returns to step 186 and user adjusts the sceneconstruction, for example by moving objects in the scene or changing thecapture device position, and recaptures the adjusted scene. If recaptureis not required (step 192) but the pixels per unit length is notsufficient (step 194), based on the analysis performed by the serviceprovider (step 206), the high resolution image is sent to the serviceprovider for analysis (step 196). The service provider performsmeasurement resolution image processing and analysis and determines anestimate of the product cost (step 208). The measurement image is stored(step 210).

Once the pixels per inch obtained from the image is sufficient (step194), the image is stored (step 198) and the user is asked whether theprocess is to be repeated for additional windows (step 200). If there isan additional window is to be processed, the method returns to step 184.Otherwise, the user places the order (step 202) and the service providerprovides dimensions for the product and forwards the order with thedimensions to the designer and/or fabricator (step 212).

A determination of whether the entire high resolution image is sentand/or analyzed may be made based on analysis of reference object pixelsper inch of the triage resolution image as described below.

Flow diagrams illustrating an example triage resolution imaging methodare shown in FIGS. 8A and 8B. The image processing of the downsizedimage may result in different feedback to the user. If image processingof the downsized image finds that all necessary criteria are met, theuser may be automatically notified that (1) product ordering may beperformed, optionally along with a cost estimate, (2) that the image maybe stored for future use, or (3) that another window may be imaged. Ifthe image processing of the downsized image finds that at least onecriterion is not met by the image, the user is notified that the scenemust be imaged again. Further, depending upon the criterion or criteriathat were not met, the user may be automatically notified of suggestedchanges to the scene or its capture that would rectify the problem. Suchsuggestions are described in more detail infra.

The automatic triage resolution image processing flow shown in FIGS. 8Aand 8B is for images containing objects upon which fiducial patternshave been printed by the end user following instructions provided by theservice provider. Prior to this automated image processing, the end usermay have provided metadata (step 220) regarding, for example the windowtype and the product that the end user is interested in potentiallypurchasing, and the end user will be provided instructions on the choiceand placement of reference and ancillary printed objects.

After capturing the image with an image capture device havingcommunication capability, the image is downsized, preferably keeping allcolor information, and along with the metadata input to the imageprocessing software. Since the service provider knows the fiducialpattern provided on the reference object, a complex pattern findingalgorithm, for example template matching using a matching metric such asnormalized cross correlation or normalized cosine coefficient, orsimilar method suited to the pattern may be used to find the referenceobject (step 222). This may be accomplished using template matchingsoftware. An example suitable for use with the present invention is thematchTemplate function available from Open Source Computer Vision(OpenCV, www.opencv.org), which is an open source computer visionrepository.

Alternatively, the user may obtain a close-up image of the referenceobject with fiducial and use software that may utilize resizing,rotating, projecting, use of Gaussian or Laplacian multiscale cascadetwo-dimensional Haar wavelet responses, integral images or othermanipulations, such as those used in the art, as in scale-invariantfeature transform or speeded up robust features object detectionsoftware.

If the reference object orientation fiducial is not detected (step 224)then feedback is provided to the user (step 256) to correct the sceneand recapture the image. If the reference object orientation fiducial isdetected (step 224), the reference object orientation is detected (step228) and if correction is required (step 230), the image orientation iscorrected using a reference object fiducial (step 258) that has, forexample, an n×m chessboard pattern of light and dark squares, where n>mand one of n and m is odd and the other is even. The n×m patternorientation may be determined by examining the coordinates of the(n−1)×(m−1) inner corners using the OpenCV functionfindChessboardCorners. Note that in this embodiment, n is assumed to beodd and m is assumed to be even (alternatively, m may be odd and neven).

This orientation is ambiguous, but the ambiguity may be resolved byexamining the upper left 2×2 orientation, which may be determined usingthe OpenCV function matchTemplate to distinguish between two possiblevertical or horizontal cases. Alternatively, orientation may bedetermined using a (1) training classifier, (2) user supplied metadata,(3) accelerometer metadata or (4) through end user confirmation oforientation. The software then compares the fiducial orientation withinthe image to the proper orientation according to object placementinstructions based on window type metadata.

If there is an orientation mismatch (step 230), the mismatch is used toobtain a “gravity down” image by flipping vertical and horizontal tocorrect an upside down mismatch (step 258), by transposing and flippingvertical if 90 degrees clockwise from correct, or transposing andflipping horizontal if 90 degrees counterclockwise from correct. In eachcase of orientation detection and correction, optionally the image sizemay be reduced prior to detection or correction. If the reference objectfiducial is not found in the image (step 224), automatic feedback to theuser may be provided suggesting that correct placement of the referenceobject on the window and properly focused imagery be confirmed (step256).

Once the image orientation is corrected (step 258) or if correction isnot required, the edges of the reference object are then found (step232). Once found, the reference object dimensions are measured (step234) and if the reference object is acceptable (step 236), the one ormore ancillary objects are found using a template match function (step240). If the reference object is found not to be acceptable (step 236),this is flagged for use feedback (step 238) and the method continueswith step 240. If the ancillary objects are found to be acceptable (step242), the software then attempts to locate the window pane (step 246).If the ancillary objects are found not to be acceptable (step 242), thisis flagged for use feedback (step 244) and the method continues withstep 246. If the window pane was successfully found (step 248), aperspective transform to be applied to the image is then determined(step 252). If the window pane was not successfully found (step 248),this is flagged for use feedback (step 250) and the method continueswith step 252. The perspective transform is then applied to the image(step 254) and any feedback previously flagged is provided to the user(step 256).

A flow diagram illustrating an example method for determining referenceobject dimensions is shown in FIG. 9. This method is used to determinethe reference object pixel dimensions and calibration factors (pixelscale factor, or pixels per unit length). The reference object is foundby determining the fiducial pattern characteristics, such as printedpattern type (e.g., chessboard or dots, hourglasses), features, size andnominal location relative to paper edges based on the digital file thatwas sent to the user (step 260). For example, using a 7×4 chessboardfiducial pattern, the feature grid is found using a method such as theOpenCV function findChessboardCorners (step 262). The local coordinateaxes are then defined (step 264) and regions of interest (ROIs) areselected for reference object edge lines (step 266).

Once the object is located, the edges are located using the relationshipbetween locations of points of the fiducial pattern in the image and thenominal locations of those points on the page in the digital file.First, the slope and length of the horizontal line along the top row ofthe 6×3 set of internal corners is found. This is done by taking thelist of 18 approximate corner locations, identifying the top row,defining regions of interest containing the upper left and upper right2×2 chessboard patterns, locating their centers more accurately by usingtemplate matching, template correlation or other complex pattern findingalgorithm with one color record (e.g., green) and subpixel offset of thebest match relative to each center to determine the top left and topright center locations (step 268). The slope of the line connectingthese points is calculated from the vertical and horizontal offsetsbetween the points and a preliminary pixels per inch value (i.e. pixelscale factor) is calculated using the ratio of the measured distance inpixels and the nominal distance between the points in inches (step 270).The calculated slope is used to determine a local rotation of thereference object and local coordinate axes.

The edges of the reference object are found by finding the center of a‘t’ using the midpoint between the top row corners, placing the crossbarat the desired vertical position and adjusting the center for smalltilt. The centers of the four edge regions of interest (ROIs) along thelines of the ‘t’ are located using the approximate pixels per inch andslope value from the ‘t’ cross point. An edge map is created by matchingor correlating corresponding edge finding templates within the regionsof interest. The extreme points along the edge map are used to determinethe edge line which should be nearly perpendicular to the intersecting‘t’ line for acceptable edge finding and measurement. Points ofintersection of the lines of the ‘t’ and the edges are determined. Thedistance (in pixels) between the left and right (H_(ref)) and the topand bottom (V_(ref)) of the reference object are based on the lineintersection points. These are used to calculate the pixels per inch ineach of the horizontal (C_(h)) and vertical (C_(v)) directions usingEquations 1 and 2, respectively, where H_(refphys) and V_(refphys) arethe reference object physical dimensions in the horizontal and verticaldirections, respectively.C _(h) =H _(ref) /H _(refphys)  (1)C _(v) =V _(ref) /V _(refphys)  (2)

An approximate reference object bounding box is calculated which isadjusted for tilt using the calculated pixels per inch and the adjustedbox is wrapped around the reference object (step 272). Once thereference object is found, a contour surrounding or adjacent to thereference object may be confirmed to be a pane or contain part of apane. The reference object location may also be used to mask thereference object, for example between finding a threshold and applyingthe threshold to the image.

After finding the reference object, ancillary objects are found. A flowdiagram illustrating a method for finding an ancillary object is shownin FIG. 10. One or more ancillary objects on which fiducials,distinguishable from that on the reference object, are printed are alsofound using template matching, template correlation or other complexpattern finding or object detection algorithms. To find the ancillaryobject(s) with fiducial pattern(s) having a pattern such as hourglass,chessboard, or hexagonal or octagonal design of alternating light anddark triangles using template matching, a corresponding template iscreated (step 280) and used to generate a search image wherein extremevalue(s) of the cosine coefficient normed metric correspond to fiduciallocations (step 282). The extreme values are then used to determine athreshold (step 284) that can be applied to the search image to mask outweak signals arising from scene content and produce an image whereinblobs of points surrounding the local extremal values remain, whereblobs, as is known in the art, are connected collections of pixelssharing a common characteristic, such as in this case, having a metricvalue above a threshold (step 286). At least one blob detector iscreated (step 288) with the desired properties of minimum blob area andseparation for finding these locations (step 289).

A flow diagram illustrating a method for finding a collection ofancillary objects is shown in FIG. 11. This method is used to find acollection of ancillary objects. The number and location of ancillaryobjects may vary such that various steps of this illustrative method areoptional and may be included depending on the particular implementationof the invention. To find the ancillary objects, regions of interest(ROIs) are first created above, below, to the right and to the left ofthe reference object (step 290). A bottom ancillary object is then foundby applying the ancillary object finding method (FIG. 10) to the ROIbelow the reference object (step 292). A top ancillary object is thenfound by applying the ancillary object finding method (FIG. 10) to theROI above the reference object (step 294). Any side ancillary object isthen found by applying the ancillary object finding method (FIG. 10) toROIs on either side of the reference object (step 296). A frame casingancillary object fiducial is then found using the side ROIs (step 298).When the frame casing ancillary object fiducial is found, the subpixelfiducial center is then located and an object bounding box is createdfor each fiducial found (step 299). Any of these ancillary objects areacceptable if their geometric properties are within specified rangesrelative to the location, size, and/or orientation of the referenceobject.

In addition, the ancillary object and/or the reference object may haveprinted a pattern that enables detection of poor focus or cameramovement at the time of capture using modulation or edge strengthmeasurements, such as a Siemens Star. If camera movement is detected byfinding directional blurring, suggestions for inhibiting movement in are-capture of the scene may be provided. If poor focus is detected byfinding general blurring, the software may suggest ways to inhibitmovement during capture, making sure the camera finds focus beforecapturing and if focus problems repeat may suggest using a differentcapture device.

In a preferred method, the user will have been instructed to place theancillary objects in regions relative to the reference object and thedark pane and the ancillary object's fiducial pattern may be, forexample, an hourglass or chessboard distinguishable by array size fromthat on the reference object. The image is searched by template matchingor correlating the corresponding template over the region(s) of interestto create a search image. For example, two ancillary objects on whichhourglass shaped fiducials are printed may be placed horizontally belowand vertically to one side of the reference object. Such ancillaryobjects may have an edge that is highly contrasting to the sash andframe to allow the contrasting edge to be easily found for subsequentmeasurement of sealing interface locations or inward facing edgelocations.

Other useful ancillary object locations include at the frame casing edgein region of interest to the side of the reference object and/or withone or more edge abutting the top of a pane for cases where anon-movable obstruction, such as a window treatment, may be present.Such ancillary objects will each contain multiple fiducials along itslength. Those that are visible below the obstruction are located and maybe used to calculate the position of the obstructed edge using therelationships between the most widely separated fiducial patterns on theobject in pixels and their known separation on the object to define ascale, finding the visible lower edge using a light/dark template andtemplate matching, and using the scale and known object dimension toestimate the location of the obstructed edge at the top. Alternatively,because such objects have been aligned with the top edge of the pane,the bottom edges or equivalently the bottom fiducials may be used todefine a line near and parallel to the top of the pane for subsequentanalysis and used to aid the pane finding and perspective transformdetermination methods described below.

If there is a mismatch between the number and/or location of ancillaryobjects found by the automated method and the user metadata expectednumber and location of ancillary objects, a message may be sent to theuser with suggestions for re-capturing the scene so that the ancillaryobjects in the scene match the metadata. When found, the subpixelfiducial centers on each ancillary object are found and an ancillaryobject bounding box is determined using the relationship between thesecenters in the image and the nominal dimensions of the object. One useof an ancillary object is to aid in the location of an obstructed paneedge, such as the top edge of a pane covered by a window treatment asdescribed below.

A flow diagram illustrating a method for finding window panes is shownin FIG. 12. As described above, the captured image may optionally beresized to triage resolution (step 300). A bilateral filter may be usedto denoise the image (step 302) and the image is made grayscale (step304). The dark parts of the image are found by performing binarythresholding (step 306). Such thresholding may be performed by applyingone or more of Huang, minimum, intermodes, fixed threshold, adaptivelocal threshold such as OpenCV adaptiveThreshold, or using variable ordynamic thresholding by first calculating the gradient and searching inthe subsection of image pixels. The binarized image resulting fromthresholding will contain one or more connected components correspondingto dark portions of the scene.

The edge of each component may be traced to provide a closed contourthat contains the component using, for example, the findContoursfunction of OpenCV (step 308). Dark portions identified in this way areexamined to remove those that are too near an image edge, too small ortoo large relative to the image area, too narrow in its aspect ratio, ortoo irregular in shape. A dark portion is deemed to be sufficientlyregular in shape (“sufficiently rectangular”) if the locations of itsedges are tightly enough distributed, ignoring small protuberances andinvaginations as well as ignoring areas known to contain reference orancillary objects. For example a histogram of the left, right, top andbottom locations of a contour can be computed and the ratio of the areasof rectangles with vertical and horizontal edges passing through the25^(th) percentile (outer box) and inner 75^(th) percentile (inner box)can be compared using an upper bound threshold to measure the departurefrom an ideal value of one attained by a non-rotated rectangle.

In the case where the problem of non-rectangularity is along the top,the portions of the contour that are above the line determined by thetop ancillary object may be replaced by points along the line. Thosecontours are judged to be sufficiently rectangular (step 310). Thesufficiently rectangular pieces form a collection around which abounding box is found for the dark pane area (step 312). If sufficientlyrectangular dark portions of the image are not found, an alternatethresholding method may be tried, and if no alternate succeeds, feedbackmay be provided to the user that the scene should be imaged when thepane background is dark or that objects may be obstructing the pane thatshould be removed from their obstructing positions.

In some instances, reflections, features of the sash holding thewindowpane or features to the exterior of the window pane (e.g., stormwindows) may lead to more than one detected image edge near the actualwindowpane edge in the image. In such cases, it may be helpful toidentify these edges and send to the user, service provider or designeran image defining choices for such edges. After human viewing of theimage with defined edge choices, the user, service provider or designermay provide input with the best edge choice for the windowpane. Forexample, preferably after applying the perspective transform describedbelow, thresholding may be performed as described above.

Template matching, using a black/white template, may be applied in thepane edge region to generate a template matching criterion curve thatcontains peaks for high value matches to the template. Such peakscorrespond to edges that may correspond to the actual pane edge in theimage. For each peak of the template matching criterion curve, a linehaving a unique characteristic to each peak (e.g., color, dash, etc.)may be respectively drawn on the image. Each line may be made visible tothe user, service provider or designer properly aligned with a pane edgein the image thus allowing choice of the correct pane edge by the personviewing the image. The person viewing this image may then provide theirchoice for the correct pane edges as metadata to the service providerand the choice may be used in subsequent steps of the image processing.

Flow diagrams illustrating a method for determining a perspectivetransform are shown in FIGS. 13A and 13B. The bounding box of the windowpane is first located (step 320). Regions of interest are then definedalong the top, right, left and bottom edges of the bounding box aroundthe dark pane area (step 322). For cases in which the true top edge ofthe pane is obscured and the top edge is defined synthetically (step324) as described above, processing of the top region of interest is notperformed and the synthetic top line is used instead of processing thetop region of interest to obtain an edge line as described below. Edgeimages of the regions of interest are then created using an edgedetector, such as a Canny edge detector (step 326). The parameters usedfor the Canny edge detector may be obtained by serially obtaining edgeimages with different parameters and examining them for suitability. Thereference object fiducial may be used to provide image by image tuningof the Canny parameters and the parameters may be different for the highand low resolution images of the same scene. Edges corresponding toreference or ancillary objects may be removed from the edge image.

A probabilistic Hough transform locates edges of interest and aconsensus edge is created from the edges of interest along each of thefour sides of the bounding box (step 328). The consensus edges aregenerated from the edges of interest (step 330). The intersection pointsof the consensus edges are used as the source points for determinationof the perspective transform using a method such as that used in OpenCVgetPerspectiveTransform (step 332). The destination points for theperspective transform are the corners of the tightest rectangle havingvertical and horizontal sides that encompasses all of the source pointsor preferably a rectangle with vertical and horizontal sides passingthough the midpoints of the consensus edges (step 334), as shown in FIG.14 where dashed box 352 represents the corner constraint box and dottedbox 354 represents the midpoint constraint box drawn around the image350 (solid box). The perspective transform based on either of these setsof destination points may not preserve the aspect ratio of objects inthe scene, nor will the overall size of objects or the location of thecenter of the image be preserved. It is desirable to modify the 3 by 3perspectivity matrix representation of the transform to accomplish theseends. First the triage image is transformed using a method such as thatused in OpenCV warpPerspective to obtain a perspective corrected triageimage. It is preferred to map destination pixels to source pixels toavoid artifacts. Interpolation may be performed by resampling usingpixel area relation. These properties are available in the OpenCVwarpPerspective function.

The reference object is then relocated in the perspective correctedtriage image, and measured to determine the pixels per unit lengthvalues as described in the discussion of FIG. 9 above (step 336). In thecase these values are not identical, the first row of the perspectivitymatrix may be multiplied by the ratio of the pixels per unit length inthe vertical and horizontal directions to balance the rows in a way thataspect ratios would be preserved (step 338). Once the rows are balanced,the overall scaling of the transformation may be determined usingmethods described in detail below wherein relationships between atwo-dimensional perspectivity matrix and a related three-dimensionalperspectivity matrix are discussed. In that discussion, a method bywhich an overall scaling parameter s may be determined is described. Thehomogeneous coordinate h_(c) of the image center is determined bymultiplying the third row of the perspectivity transform and thecoordinates of the image center. The first two rows of the balancedperspectivity transform are then multiplied by h_(c) and divided by thisvalue of s to normalize the transform so that the overall scalingparameter is 1 (step 340).

Finally, a shift in the location of the image center is determined byapplying the balanced and normalized perspectivity matrix to thelocation of the image center in the triage image to obtain its positionin an image that would be obtained using that matrix using a functionsuch as OpenCV perspectiveTransform. This shift is used to adjust thetranslation vector located in the first two rows of the third column ofthe perspectivity matrix by subtraction so that the location of theimage center is preserved (step 342). This balanced, normalized andcenter preserving transform becomes the final triage image perspectivetransform. The determined transform parameters may be used toautomatically decide whether the scene should be re-captured andguidance provided to the user to move closer to the projection of thewindow center for the re-capture of the scene using methods described indetail below. After the transform is determined, repeated correction ofthe perspective distortion in the triage resolution image is optional.As described below, the triage image perspective transform may be usedto transform higher resolution images prior to steps needed formeasurements. While the above description utilized the pane bounding boxto determine source points for the perspective transform, the referenceobject bounding box obtained in step 272 may be used for this purpose.

A flow diagram illustrating an example preparation for measurementmethod is shown in FIG. 15. The method of measuring target objects inthe scene is shown in FIG. 16. With reference first to the method ofFIG. 15, to prepare for the measurement the triage image is uploaded(step 360) and analyzed as described in FIGS. 8A and 8B (step 362). Thisanalysis provides a measured value for the pixels per unit length in theplane of the reference object. This triage value should be chosen to beat least 10 pixels per inch, preferably at least 15 pixels per inch,more preferably at least 30 pixels per inch. If the chosen triage valueis exceeded by the measured value (step 364) in the analysis of thetriage image, transmission of the high resolution image may be abortedand the triage image may become the Provisional Measurement Image (376)after applying the perspective transform to the triage image (step 366).If the triage value is not exceeded (step 364), the higher resolutionimage transmission is completed (step 368).

The high resolution image orientation is determined and, if necessarycorrected (step 370). The first perspective transform determined, forexample as shown in FIGS. 8A and 8B, for the triage image is rescaled(step 372), for example by using a matrix similarity transform to adjustfor scale difference, and applied to the high resolution image (step374) to form the Provisional Measurement Image (step 376). SuchProvisional Measurement Image has vertical and horizontal features inthe scene that are substantially, although not necessarily exactly,vertical and horizontal in the image. Of particular importance, removalof the bulk of perspective distortion results in more accuratemeasurements of the reference object.

One embodiment for image processing workflow may be completed as shownin FIG. 16. This workflow employs results from the triage resolutionanalysis, including, for example, object (reference object, ancillaryobject and window pane) locations, the perspective transform andmetadata (step 380). It is important to note that even after allperspective distortion is removed from the scene, the reference andancillary objects may still be rotated relative to the window due tomanual placement variability and may contain some aspect ratiodistortion after the perspective correction is applied. The transformcan also be applied to coordinates of the locations of objects in thescene from the triage resolution analysis to aid in determining smallerregions of interest for efficient location of these objects in theprovisional measurement resolution image, for example, to find referenceand ancillary objects and window panes.

Using location information from the triage resolution image analysis andfollowing the method described for the triage resolution image workflow,the reference object is found and its dimensions measured (step 382).Similarly, the ancillary objects and windowpane are found using locationinformation from the triage resolution image analysis and the methodsdescribed for that analysis (step 382). When the pane edge locations arefound in the image using the same light/dark template matching method asused to determine the edges lines of the reference object (step 384),the slopes of these edge lines defined in such a way that the verticallines have zero slope along the sides and horizontal lines have zeroslope along the top and bottom are also recorded.

These slopes are then tested to determine if the transformed image ofthe pane is rectangular and aligned to the image axes (step 386). If theslopes are all the same to within a determined tolerance (step 388), thepane is determined to be rectangular. If the slopes are all zero towithin a second determined tolerance, the pane is determined to bealigned. If both of these conditions are met, an improved transform isnot needed, resulting in enhanced performance by avoiding the need tocompute and apply a second transform and relocate the objects and pane(step 390). This is typically the case in many images.

If either of the above conditions is not met for the first time (step400), a final transformation may be determined using the edge locationsand slopes to directly define source and destination points analogouslyto the method used to determine the transform in the low resolutionworkflow (step 404). This transform may then be applied to the firsttransformed provisional measurement resolution image to obtain a finaltransformed image, the measurement image, and recomputing the object andpane finding steps. The rectangularity and alignment tests may bereevaluated and, although it is possible to repeat the loop uponfailure, typically at most a single final correction is required.Therefore, the potential failure is noted for review and the methodcontinues as if it had succeeded (step 402). When the measurement imageis ready, it may be downsized and stored if desired.

For ancillary objects described above, the outward high contrast edgelines associated with them are found and their locations determined toallow locating target objects or features of interest with respect tothe nearest parallel pane edge (step 392). These locations may be foundusing template matching or correlating with a template that is dark inone half and light in the other with template orientations chosenaccording to the edge direction and darkness/lightness pattern ofinterest. Locations of these features may be determined using techniquessuch as dark/light template edge matching or correlating. Windows mayinclude additional features of interest such as muntins or grids, checkrails, or mullions that are easily locatable in the scene given paneedge and ancillary object locations (step 394). While the above methodprovides for a full automation of measurement, the measurement image mayalso be provided to the end user and/or designer, optionally withidentification of ancillary object edges of interest, so that the targetobject edge locations can be confirmed or adjusted by the end user ordesigner.

Using the pixels per unit length calculated when finding the referenceobject in the measurement image, the pane dimensions for the pane onwhich the reference object resides may be directly calculated from thepane pixel dimensions in the measurement image (step 396). Suchdimensions may be used to directly specify a pane related product or maybe combined with other measurements to specify other products that mayrelate to sealing interfaces, inward facing frame dimensions or framecasing related products including pane related and other such productsas described in more detail in co-pending U.S. patent application Ser.No. 14/315,503, to Wexler et al., entitled “Supplemental Window ForFenestration”, filed Jun. 26, 2014, incorporated herein by reference inits entirety.

A flow diagram illustrating an example method for calculating a targetobject dimension substantially in the same plane as the reference objectis shown in FIG. 17. When the user metadata indicates that a productrelating to, for example, sealing interfaces or inward facing framesurfaces is desired, the dimensions of the window pane on which thereference object resides is calculated (step 410) as described above.The pixel distance from the side ancillary object outer vertical edgethat identifies the sealing interface or inward facing frame surfacelocation to the nearest vertical pane parallel edge is determined (step412). The pixel distance determined in step 412 may be converted to aphysical distance using the horizontal pixels per inch to obtain thehorizontal distance from the pane edge to either the sealing interfaceor inward facing surface (step 414). Such a distance may be useful inspecifying auxiliary parts as described in U.S. patent application Ser.No. 14/315,503. The pixel distance determined in step 412 is thenmultiplied by two (taking advantage of symmetry) and added to the panehorizontal pixel dimension, the sum of which is converted to ahorizontal physical dimension using the horizontal pixels per inchcalculated when finding the reference object for the measurement image(step 416). Similarly for the vertical dimension, the pixel distancefrom the bottom ancillary object outer horizontal edge to the bottompane edge is determined, multiplied by two and added to the panevertical pixel dimension found for the measurement image. This isconverted to a physical dimension using the vertical pixels per inchcalculated when finding the reference object for the measurement image.

A flow diagram illustrating an example method for calculating the targetobject dimension offset from the reference object plane is shown in FIG.18. For dimensions that are not substantially in the same plane as thereference object, fiducial size differences relative to the referenceobject (step 420) may be used to make adjustments needed to account forout-of-plane scale distortions induced by the use of the pane planeprojection transform and may be used to calculate corrected pixels perinch for the plane of interest to allow for more accurate measurement ofsuch dimensions. These out of plane distortions include bothmagnification, with objects in closer planes appearing larger andfurther planes appearing smaller, and translation, with objects shiftingtheir position according to their distance from the camera location intheir plane as shown in FIG. 19, using methods based on geometric scenereconstruction described below.

The relative pixels per unit length (i.e. plane offset scaling factor)for a fiducial in an arbitrary plane and the reference object on thepane in the transformed measurement image (step 422) together withgeometric scene reconstruction also allows estimation of the distancebetween the planes, for example the distance between the planescontaining each pane of a vertical or horizontal sliding window (step424). These methods may be used, for example, to locate edges of awindow's frame casing when a fiducial is used to identify theoutward-most frame casing edge (step 426). The symmetric horizontaloffset of the outer frame casing on each side of the pane is used tocalculate the outer frame casing horizontal dimension using themeasurement image pane dimension and scaling factor (step 428). Thesemethods may also be used to locate points on a single or double hungwindow top sash on which a fiducial containing ancillary object has beenplaced, to correct the locations of various points in the top sash paneplane. Symmetry and/or machine learning may be used as an alternative tothis approach for the vertical sliding window top (exterior) sash whenthe reference object is placed on the bottom (interior) sash. Withproduct specification calculated, a corresponding product cost may bedetermined and provided to the end user, for example in a digitalshopping cart. In addition, the dimensions of the specified product maybe provided to a product designer and/or fabricator.

It should be noted that a camera calibration model that allowscorrection of other distortions due to lens design and sensor to lensalignment may be obtained by analysis of several images of asufficiently complex pattern that covers a sufficiently large portion ofthe image. Such models are typically incorporated into the camerasoftware to obtain distortion free images and are trivially added as apreprocessing step if a camera is discovered to have such distortions.In the description below, it is assumed that such a camera calibrationmodel has been applied to obtain the original image into which areference object has been placed. In the above described methods theperspective transform derived using coordinates of pane corners toestablish source and destination points does not in general preserveaspect ratios of physical objects in the plane of the pane. Thesedistortions require the use of two separate scaling factors forconverting vertical and horizontal displacements in pixels into physicalunits. A preferred method, in which the transform does preserve aspectratio and which has other advantages, will now be described.

The substantially planar reference object comprising a pattern havingfour or more locatable points with nominal positions and recognizableusing image analysis methods described above, such as chessboard cornerfinding, dot grid center finding, QR barcode finding or similar patternfinding is placed in the scene and found as described above. The patternon the reference object contains information sufficient to determinegeneral image orientation and a combination of transposition, verticalflip or horizontal flip operations may be used to correct theorientation as described above.

The locatable points are measured and the measured locations and nominallocations are used to derive a nominal printed pattern mapping in theform of a 3×3 projectivity matrix, also known as a perspectivetransform. This derivation may be performed using, for example, themethod used in OpenCV getPerspectiveTransform for four points orfindHomography for four or more points. Such a mapping defines arelationship between nominal points in the plane of the reference objectand points in the image. When image locations are used as source pointsand nominal locations are used as destination points, the derivednominal inverse mapping would be from image point locations to nominalpoint locations. Such a transformation is invertible to form a nominalcapture mapping and together with the nominal inverse mapping allowsmapping points between the nominal reference object plane and the imagein either direction. A capture mapping can be found by switching theroles of source and destination in the routine used to derive thetransform.

A nominal scaling between image distance units in pixels and nominaldistance units is easily obtained by applying the nominal capturemapping to a pair of nominal points that are one unit apart andcomputing the distance between the points in image pixel units. Such apixel per distance scale factor can be applied to the nominal captureand inverse mappings by scaling the top two rows of the inverseprojectivity matrix by the scaling factor and scaling the top two rowsof the capture projectivity matrix by the inverse of the scaling factorto obtain projectivity matrices that may be used to apply to digitalimages represented by code values at pixel coordinates to obtainresulting digital images using digital imaging software such as OpenCVusing its warpPerspective function without dramatic changes in overallresolution.

The reference object edges are then located in the image as follows. Ascaled nominal inverse mapping may be applied to an entire image, forexample using OpenCV warpPerspective, to obtain a nominal image in whichthe reference object pattern matches the nominal geometry in pixelscaled units. If the physical pattern locations match the nominalpattern locations exactly on the reference object and the mapping isexact, the edge locations in the nominal image are completelydetermined. When such a pattern is printed on paper, the printed patternmay be mispositioned on the page relative to its nominal geometry byscaling in one or both directions as well as translation on the page dueto printer driver settings and further misplacement plus rotation withinthe plane due to paper to printer transport biases. Such misplacementsare typically small, but lead to an uncertainty in the locations of theactual edges of the reference object in the image relative to theprinted pattern.

In addition, the transformation itself is subject to errors due tomeasurement errors in the locations of the points used to define it.Therefore the edges of the printed pattern need to be located moreprecisely in order to define a relationship between physical locationsof points in the plane of the reference object and points in the image.This may be accomplished by establishing locations in the nominal planewhich contain the edges with a high degree of certainty, using thenominal capture mapping transform to determine approximate locations ofthese edges in the image, surrounding these image locations withsufficient additional height and width to create regions of interest inwhich an edge detection method may be applied, and then applying theedge detection method to define lines which match the location of theedges in original image coordinates.

Preferably, the scaled nominal inverse transform is applied to the imageor a portion of the image which contains the aforementioned regions ofinterest to create an image in which the edges are substantiallyvertical and horizontal and create regions of interest along each edgewhich contain the edge with a high degree of certainty and allow for useof an edge detection method to define lines which match the location ofthe edges in the scaled nominal image coordinate system. Regardless ofthe coordinate system in which the edges are located, the appropriateinverse or capture transform may be used to locate them in the other asneeded.

The physical transforms may now be determined. Once the edge lines arelocated, a relationship may be developed which characterizes therelationship between nominal and physical locations of printed patternson a reference object as well as a relationship between original imageand physical locations. Four edge line locations may be used to inferlocations of the corners of a rectangular reference object in thecorresponding original image or nominal image coordinate systems. Thesemay be used along with the physical dimensions of the reference objectto define a perspective transform. If the edge lines are defined in thenominal printed pattern coordinate system, a Nominal to Physical modelis obtained. This transform may be composed with the Image to Nominaltransform to obtain an Image to Physical transform. Alternately, if theedge lines are defined in the original image coordinate system, an Imageto Physical model is obtained. This model may be composed with theinverse of the Image to Nominal transform to obtain a Nominal toPhysical transform. Again, to avoid drastic changes in resolution whenapplying a transform to images to produce an image, the Image toPhysical transform is preferably scaled to produce results in pixelunits.

The Image to Physical transform allows transformation of the originalimage into an image in which distances within the plane of the referenceobject that are measured in pixel units are directly proportional tophysical distances by a pixel per physical unit ratio. Additionally, theaspect ratio of rectangular objects in this coordinate system ispreserved for a rectangular object in any plane parallel to the plane ofthe reference object. The Nominal to Physical transform can be derivedin matching units to allow transformation of other printed patterns intophysical coordinates, enabling establishment of their physical size fromnominal sizes, a feature useful in establishing sizes of other printedobjects when they do not include known physical dimensions, such asprinted patterns that are cut out of a page using cutters, scissors, oreven fold and tear methods.

The image may now be transformed into physical coordinates. In thisstep, we apply the resolution preserving scaled image to physicalinverse transform to the image. Preferably, this is achieved by inverseinterpolation using the transform to determine the location in theoriginal image corresponding to a location in the physical image andinterpolating into the original image using these coordinates. This maybe accomplished, for example, using the warpPerspective function ofOpenCV.

Other objects placed in the scene may be located as described above. Inthose methods, the nominal dimensions of the patterns in the objectdesign are used to aid in locating and measuring the patterns whileanalyzing images. These nominal dimensions may be converted to physicaldimensions using the Nominal to Physical transform to obtain dimensionsappropriate for use with an image in scaled physical units.

When the reference object is placed in the scene in a way that it issubstantially parallel to the plane of the windowpane, it may be rotatedrelative to the natural windowpane coordinate system which we refer toas the world coordinate system. Location of a single edge of the pane inthe physical image allows determination of this rotation and therotational aspect of the physical transform in the upper two rows of thetransform could be modified to obtain a final by multiplying by arotation matrix that results in the pane edge being vertical orhorizontal as desired. As all four edges are to be located in any case,the average rotation of the edges may be used along with the spread ofthe rotations to determine whether the only remaining effect is thisrotation or whether there is some residual distortion that requires afurther perspective correction. This correction will result in a verymodest apparent aspect ratio distortion in the reference object as wellas a very modest change in the scale factor derived from a unit physicalvector that may be corrected by scaling corresponding rows in theperspectivity matrix.

In the present invention, there may be instances in which measurement ofdimensions in parallel offset planes is desirable. For example, slidingor hung windows have panes in parallel planes that require measurementor the frame casing around a window may be in a different plane than thepane on which the reference object is placed. Also, the wall in which awindow is mounted may be in a parallel plane offset from the referenceobject plane. In addition, there may be custom supplemental products forwhich complete specification requires a dimension estimate perpendicularto the plane of the window, such as described in U.S. Pat. No.8,923,650.

Using the reference object and ancillary objects, for example asdescribed above, in offset planes, 3D projective geometric modelingtogether with information regarding the relationship between camera lensfocal length and sensor size, typically described using a 35 mmequivalent focal length, may be used to analyze the 2D images obtainedin the present invention to provide measurement estimates between andwithin the offset planes. In addition, such modeling may be useful indetermining whether the image was captured from within the interiorprojection of the window. Although it is technically possible todetermine a pose and position of the reference object or pane relativeto the camera by analysis of the projectivity matrix, this commonlyperformed geometric scene reconstruction task may be accomplisheddirectly using routines such as solvePnP provided in OpenCV given thesame source and destination data together with camera intrinsicparameter data. The routine provides a rotation vector r and atranslation vector T that fully describe the relationship between theset of world and camera coordinate systems. The rotation vector may beconverted into an orthonormal rotation matrix R or vice-versa using theOpenCV Rodrigues function, which together with the translation vectormay be used to form a set of linear equations, as shown in Equation 3below.

$\begin{matrix}{\begin{bmatrix}x \\y \\z\end{bmatrix} = \left\lbrack \begin{matrix}{sR} & {\left. {sT} \right\rbrack\begin{bmatrix}u \\v \\w \\1\end{bmatrix}}\end{matrix} \right.} & (3)\end{matrix}$

The camera information required to obtain the translation vector betweenthe camera origin and the world origin includes both the focal length ofthe camera and the location of the camera axis, referred to as thecamera principal point, in the image. One may use the center of theimage in pixel units measured from the upper left corner as theprincipal point and define the focal length of the camera in pixels aswell. The ratio of the length of the diagonal of the image in pixels tothe focal length of the lens in pixels equals the ratio of the diagonalof a 24×36 mm rectangle to the 35 mm equivalent focal length of thecamera. In addition, if these ratio values were known, the distance ofthe camera to the origin of the world coordinate system could bedetermined using this ratio and a known diagonal length. Informationregarding these ratio values is publicly available from various sites onthe internet and could be used to choose a value given cameraidentification information. A 35 mm equivalent focal length is oftenincorporated in image EXIF metadata. In cameras used in smartphones, therange of values of 35 mm equivalent focal length is typically within a28-35 mm range, with many at or near 30 mm, and we have found that useof an approximately 30 mm assumed 35 mm equivalent focal length iseffective at providing a camera focal length. When the solvePnP routineis provided these camera parameters together with the source anddestination points in the camera image and world coordinates, theresulting rotation and translation information is in the cameracoordinate system.

The translation in the world coordinate system may then be determined bymultiplying the inverse of the rotation matrix, namely its transpose,times the translation vector and negating the result to obtain the worldcoordinates of the camera location both within the plane of the windowas well as the camera-to-subject distance from that plane. This locationmay then be compared to the coordinates of the window boundaries todetermine if the image was captured from within the orthographicprojection of the window. Further, once a camera to subject distance isknown, relative magnification of measureable objects in planes parallelto the plane of the reference object may be used to determine theirdistance offset along the camera axis which may be resolved intocomponents in the world coordinate system, enabling estimation of planeto plane offset in world coordinates as well as their translation in theplane coordinates of the image after the inverse mapping is applied.

When doing projective geometry, a fourth row is added to account forambiguity of locations of points in the real world along lines throughthe camera coordinate system origin, resulting in a model usinghomogeneous coordinates wherein points [x y z l], [hx hy hz h] and soforth are all regarded as identical. The model equation relatingcoordinates [u v w] and [x y z] is now nonlinear, but can be expressedusing a linear set of equations encapsulated into a 3D projectivitymatrix followed by rescaling using the fourth coordinate as shown belowin Equation 4:

$\begin{matrix}{\begin{bmatrix}{hx} \\{hy} \\{hz} \\h\end{bmatrix} = {\begin{bmatrix}{sR} & {sT} \\H & 1\end{bmatrix}\begin{bmatrix}u \\v \\w \\1\end{bmatrix}}} & (4)\end{matrix}$

The 2D model for the pane plane is obtained by deleting the thirdcolumn, as w=0, and the third row since we are concerned only with theimage plane. A 2D transform that applies to another plane parallel tothe plane of the pane is related to the 2D transform for the plane ofthe pane by changes in the offset T and the scalings s and h that dependon the distance between the planes, the third entry in H and the thirdcolumn of R. A general form valid for any w is given by Equation 5below:

$\begin{matrix}{\begin{bmatrix}{h^{\prime}x} \\{h^{\prime}y} \\h^{\prime}\end{bmatrix} = {\begin{bmatrix}{s^{\prime}R_{uv}} & {s^{\prime}T_{xy}^{\prime}} \\H_{uv}^{\prime} & 1\end{bmatrix}\begin{bmatrix}u \\v \\1\end{bmatrix}}} & (5)\end{matrix}$

wherein R_(uv) is the upper left 2×2 block of R and the remaining itemsare given by Equations 6 through 9 wherein H_(w) is the third element ofH, H_(uv) is a 2-vector comprising the first two elements of H, T_(xy)and T′_(xy) are 2-vectors comprising the first two elements of T and T′respectively, and R_(xy,w)) is a 2-vector comprising the first twoelements of the third column of R, as follows:h′=h/(1+wH _(w))  (6)s′=s/(1+wH _(w))  (7)H′ _(uv) =H _(uv)/(1+wH _(w))  (8)T′ _(xy) =T _(xy) +wR _(xy,w)  (9)

If a 2D model is provided that applies for w=0, this process may besubstantially reversed to create a full 3D model form in which all buttwo of the elements of the projectivity matrix are determined. The factthat the upper left 2×2 in the 2D model is a scaled copy of the upperleft 2×2 of the 3D model allows determination of the third row andcolumn using properties of a full rotation matrix R. The orthonormalmatrix R has the property that the sums of the squares of the elementsof the rows are all equal to one and that the sums of products ofelements in different rows are all zero. Since we have only two of thethree elements in each row of the 2D model and they have been scaled bythe same number, we have two unknowns to determine the first two rows ofthe scaled 3D rotation matrix. We can use the orthonormality of R togenerate two equations which determine these values to within a sign.Once these values are chosen, a scale value s can be determined as theroot sum square of either row. Finally, a third row is obtained bytaking the vector cross product of the first two rows, resulting in avector with length equal to the square of the scale value, so it may berescaled by dividing by the scale value s.

The sign ambiguity in the first two elements of the third column resultsin a paired ambiguity in the first two elements of the third row. Thesign ambiguity may be resolved by examination of the location in theoriginal image of the corners of a rectangle surrounding the imagecenter using the 3D model, choosing the sign that results in theappropriate sign of z. The distance increment from the camera, z, ispositive in areas where the corners are closer to the center of theoriginal image and negative in areas where the corners are further awayfrom the center. This model may be used for the plane where w=0 withoutdetermining the third elements of H and T, but if it is to be applied toanother plane wherein w is not zero, the model is not well defined. Forpurposes of determining a rotation vector however, the rotation matrix Ris fully defined, enabling estimation of distances as described above.

A preferred method of using geometric scene reconstruction fordetermining relative location differences for objects in a plane whose wcoordinate is not zero is now described as illustrated in FIG. 19. FIG.19 shows an orthographic view of a measurement image with a pane alignedto the horizontal u-axis and the vertical v-axis, shown as rectangle430, an out of plane ancillary object 432 containing fiducials to theleft of the pane, the location of the original image center as circle437 near the center of the measurement image, and the location of thecamera as a second circle 439 below and to the left of the center.

Note that although the inside of rectangle 430 is shown as white in FIG.19, if the photo of the window is taken at night, the interior ofrectangle 430 (i.e. the window pane) will be dark and the two smallcircles will be against a dark background.

FIG. 19 also shows a view of the scene from above (in the v direction)with the plane of the pane shown as a solid horizontal line 434 in theu-axis direction, the planes of the ancillary object shown as horizontaldashed lines 436 and central projection lines from the camera location(open circle 439) to various points of interest in the scene such as theimage center location 437, with the vertical line in the w-axisdirection. The distance between the planes is indicated along a verticaldashed line placed at the pane edge location. The location of theancillary edge of interest is shown in the ancillary plane and itslocation in the measurement image is shown in the pane plane.

The apparent distance from the pane edge to the ancillary edge ofinterest is magnified and shifted due to the projection, and means tocompute the horizontal location of the ancillary edge of interest is nowdescribed. The distance between the fiducials in pixels is measured inthe measurement image and the pixels per unit length is calculated usingknowledge of the physical distance between fiducial centers. The pixelsper unit length of the ancillary object thus measured applies throughoutthe plane. The ratio of this value to the corresponding value obtainedfrom the reference object forms a scale factor s. This scale factor isrelated using properties of similar triangles to the ratio of distancesfrom the two planes to the camera and to the ratio of distances of theancillary object from the camera position, as follows:

$\begin{matrix}{s = {\frac{{\Delta c}^{\prime}}{\Delta\; c} = {\frac{{\Delta c}^{\prime} + {\Delta\; h} + {\Delta\; p}}{{\Delta\; c} + {\Delta\; p}} = \frac{D^{\prime}}{D^{\prime} + {\Delta\; d}}}}} & (10)\end{matrix}$

In this equation, Δd is negative when the ancillary object is closer tothe camera. The amount of offset due to the projection of the offsettingdistance normal to the planes Δh is positive when the camera position isinside its location and is negative when the camera position is outsideits position as would be expected for a visible or blocked segmentrespectively. The distance of interest for the measurement of objects inthe plane of the ancillary object is Δc, and its value can be computedusing the scale factor s and values that can be measured in themeasurement image, where Δp is the horizontal distance between thecamera location and the pane edge and the sum (Δc′+Δh) is the horizontaldistance between the pane edge and the projected ancillary object edgeof interest:

$\begin{matrix}{{\Delta\; c} = {\frac{\left( {{\Delta c}^{\prime} + {\Delta\; h}} \right)}{s} + {\left( {\frac{1}{s} - 1} \right)\Delta\; p}}} & (11)\end{matrix}$

The distance of interest if a depth measurement is desired is Δd and itssigned value can be computed using the scale factor and the distance ofthe camera from the pane plane:

$\begin{matrix}{{\Delta\; d} = {\left( {\frac{1}{s} - 1} \right)D^{\prime}}} & (12)\end{matrix}$

Note that errors in estimating the location of the camera center due toany differences in the 35 mm equivalent focal length used in geometricscene reconstruction or errors in locating points in the scene whichwould lead to errors in Δp are multiplied by (1/s−1), which according toEquation 12 is the ratio of Δd to D′, a value that is typically small.

In one embodiment, when using symmetry to aid with calculatingmeasurements, a useful input to the algorithm are the vertical andhorizontal positions of the capture device with respect to the window.Such information may be obtained from the image by analysis of the imagereflected in the windowpane. For example, the software may pre-set theflash setting to fire at the time of capture creating a bright spot witha roughly circular shape, and characteristic intensity falloff withdistance within the dark pane of a frontlit scene. Based on these brightspot characteristics and, optionally, the size of the bright spot, thecapture device location may be determined. The bright spot positionprovides a very close estimate of the camera lens location at the timeof capture, particularly when a smartphone is used as the capturedevice. In this case, the distance Δp needed to properly calculate thelocation of the ancillary object using Equation 11 does not requirelocating the camera position in space, but estimates of the depth arenot enabled without geometric scene reconstruction using camera 35 mmequivalent focal length as described above.

When two locations in a plane whose w coordinate is not zero are knownto be symmetrically placed horizontally with respect to the horizontalpane center, Equation 11 may be employed on both left (L) and right (R)sides of the pane. Furthermore, since the total distance across the paneΔp_(L)+Δp_(R)=p is known and since the objects are symmetrically placedso that their distances Δc_(L) and Δc_(R) from the pane edges are equal,the following system of four linear equations are obtained relating thevarious distances:

$\begin{matrix}{{\Delta\; c_{L}} = {\frac{\left( {{\Delta c}_{L}^{\prime} + {\Delta\; h_{L}}} \right)}{s} + {\left( {\frac{1}{s} - 1} \right)\Delta\; p_{L}}}} & (13) \\{{\Delta\; c_{R}} = {\frac{\left( {{\Delta c}_{R}^{\prime} + {\Delta\; h_{R}}} \right)}{s} + {\left( {\frac{1}{s} - 1} \right)\Delta\; p_{R}}}} & (14) \\{p = {{\Delta\; p_{L}} + {\Delta\; p_{R}}}} & (15) \\{{\Delta\; c_{L}} = {\Delta\; c_{R}}} & (16)\end{matrix}$

This system of equations is easily solved to provide a horizontal cameralocation relative to the pane edge(s) as well as the distance from thepane edges of the symmetrically placed objects. If a rotation vector isobtained using the capture transform as described above, this horizontallocation in the w=0 plane is sufficient to determine the verticallocation of the camera in the w=0 plane as well as a camera distancefrom the w=0 plane without employing knowledge of a 35 mm equivalentfocal length. Alternately, if an object in a scene lies in a plane notparallel to the w=0 plane, measured distances in that object may beutilized to determine the w locations of features on that object.

For example, a rectangular object of known dimensions may be placed in aplane in which the vertical location v in world coordinates is constantand is oriented so that two of its sides have constant w coordinatesw_(R) (442) and w_(F) (444) and the other two have constant ucoordinates u_(L) and u_(R) as shown in FIG. 20. Triangle similarityrelations among the lengths of the projections of the front and rearedges indicated by dash-dotted lines 446 and the distances of theseedges from the camera location 438 at w=w_(E) given the known dimensionsu_(R)-u_(L) and w_(R)-w_(F) permit determination of the camera distancefrom the w=0 plane (440), again without employing knowledge of a 35 mmfocal length. In either of these cases, the distance from the originalimage center location in the w=0 plane to the camera location in the w=0plane together with the camera distance from the w=0 plane allowdetermination of the camera-to-subject distance. This together with thedistance across the image along the pivot axis in pixels and theconversion factor between pixel distances and physical distances allowdetermination of a 35 mm equivalent focal length for the camera whichmay be used in other scenes not containing symmetry objects or objectswith known dimensions placed in planes not parallel to the w=0 plane.

For methods that utilize fiducial patterns or additional objects placedin the scene to aid in finding target objects or locations within animage, it will be appreciated by those skilled in the art that thefiducial patterns may be printed to appear black or any other coloringand the additional objects may be of any coloring that providessufficient contrast and differentiation relative to the scene ofinterest. Colored fiducials or objects, especially those having bold,vibrant color, may be helpful to aid with finding objects placed in thescene that are to be used for measurement. Also, use of such colors maybe useful when capturing window images when a significant portion of thewindowpane has light entering from the outdoors.

A fully automated metadata gathering method may be used in which imageanalysis automatically generates metadata, for example, windoworientation based on the location of a windowsill or window treatmentattribute, used in subsequent constraint calculation. Metadata that maybe useful in the present invention includes the window orientation,window type (e.g., single or double hung, sliding, casement, fixed,etc.), and location and type of object in the image such as a referenceobject or window treatment and associated control mechanism. In additionto image related metadata, the end user may provide order information,such as payment and delivery information and preferences, at any timeduring the process prior to submitting an order. End user preferencesmay include the type of sheet including material composition, opticalproperties, the number and location of sheets and whether window orwindow shade operability is desired. Additionally, metadata orconstraint calculation accuracy may be confirmed with the end user aspart of the process, optionally using the digital image or annotatedversion of the digital image.

In another embodiment, software incorporated in the capture device, suchas CameraSharp, corrects for camera movement at the time of capture ormeasures the amount of movement and alerts the end user to capture a newimage if the movement is found to be above a predetermined threshold.The predetermined threshold may be varied depending upon the size of thereference object used or the ratio of its size to the size of the imagecaptured. Also, it is preferable to keep the exposure time as small aspossible while still capturing sufficient light to identify thereference object and constraints in the image. In one embodiment, theexposure time should is less than 0.125 second. Additionally, to inhibitthe impact of end user movement during image capture, it is preferred tominimize or remove delay between the end user shutter actuating movementand the actual shutter actuation or to use voice actuation of theshutter. Such exposures may be enabled using software that overrides anydevice manufacturer incorporated shutter actuation delay.

The digital image undergoes image processing that provides dimensionalinformation for the fenestration, frame and treatments so thatappropriately dimensioned custom supplemental parts may be designed andmanufactured for installation at the fenestration site. Morespecifically, an end user, such as the owner or renter of an indoorspace having a window or someone hired by such owner or renter, selectsa window in that space for modification to decrease optical transparencyor heat flow by conduction and/or emission through the fenestration. Theend user obtains a digital image of the selected window. The digitalimage may be obtained using any type of image capture device such as amobile device containing an image sensor or in communication with anexternal image sensor (e.g., a webcam), for example a digital still,including rapid multi-exposure digital still, or video camera, a cameraphone or smartphone, a laptop computer, a tablet computer or othermobile device.

After obtaining the digital image, the digital image and associatedmetadata undergo digital image processing. Note that in the case wherethe digital image processing occurs on a server remote from a hand heldmobile device such as a smartphone that captured the image, the digitalimage is obtained by the mobile device transmitting the image to theserver over a network such as the Internet. In the case where thedigital image processing occurs on the mobile device that captured theimage, the digital image is obtained by well known internal processorcommunications within the mobile device itself. In the case where theimage is sourced from a service provider, the digital image is forwardedover a network to the server performing the digital image processing.Such forwarding may be implemented using any well known technique suchas email attachment, ftp transfer, http transfer, etc.

The digital image processing performed on the image and associatedmetadata may occur in one or more locations depending upon computingpower and software availability as well as the extent to whichautomation is used. In one embodiment, the end user sends the digitalimage and associated metadata to a service provider. As part of themetadata provided by the end user, the end user may click or tap onlines or objects or use a crop tool to identify locations in the imageto be used for calculating constraint dimensions. The end user metadatainput may be provided using a software application that prompts the enduser for specific information that will aid in calculating theconstraint dimensions.

When custom supplemental parts for more than one window is desired bythe end user, the end user may indicate aspects of all the windows thatare to be the same so that the metadata input by the end user may beless cumbersome and redundant images may be omitted. The softwareapplication may also include image comparison capability so that similarwindows may be automatically suggested or identified. Such imagecomparison capability may include identifying windows having the nearlyidentical dimensions, framing, sash in-frame and tilt lock locations,muntin type and location, and sash handle type and location.

In one embodiment, the service provider uses digital image processingalgorithms to determine the dimensions of, for example, the window,window frame or window treatments. The dimensions are used to design,either automatically or semi-automatically, custom supplemental partsthat will fit to the window and/or frame, taking into considerationoperability of the window, any window treatment present and end userpreference. The design is then used to custom fabricate at least customsupplemental part and means for supporting such custom supplemental partso that at least a portion of the window may be covered. Alternatively,software may be used by the end user so that image processing andcalculations may be performed with the capture device. Image processingand/or calculational software may also be used by the end user, serviceprovider and/or fabricator in conjunction with a computing device, storebased kiosk or other computing devices or services such as cloudcomputing services, or any combination thereof.

In one embodiment, metadata regarding the conditions of the imagecapture at the time of digital image capture are obtained. If the deviceused to obtain or capture the digital image provides metadata with thedigital image, such metadata is used to minimize end user input ofmetadata. For example, the present invention can beneficially usestandard metadata formats such as those governed by Exif, IPTC, XMP,DCMI or PLUS. Such formats provide information that may be useful forapplying image corrections including the capture device make and model,orientation/rotation, compression, resolution, flash use, focal length,aperture value, ISO speed and pixel dimension, shutter speed andlighting.

Additional metadata provided by the end user may be provided at the timeof image capture or at another time using another digital device such asa computer, kiosk or website. End user metadata may include specificwindow information if custom supplemental parts are to be provided formore than one window. For example, a window identifier such as “Joe'sBedroom South Wall” might be used to distinguish from “Joe's BedroomWest Wall”. Such an identifier may remain with the image throughmanufacturing of the parts associated with a given window so that theidentifier may be printed or embossed on each part associated with thatwindow. Also, the end user may wish to specify what type of customsupplemental part is desired. For example, different types of plasticsheets may be used to cover a window, such as transparent,semi-transparent, opaque, tinted or low-e with variations of solar gain.The plastic sheet may have additional functionality such as a flexiblesolar cell array as is known in the art, for example as described inU.S. Pat. No. 7,675,057 and U.S. Publication No. 2012/0125419, both ofwhich are incorporated herein by reference in their entirety.

In addition, the end user may provide a manual measurement to aid in thecalculation of other dimensions. Depending upon what type ofsupplemental part is desired by the end user, different sets of mountingsurfaces may be used so the user may specify, on the capture device orother device, which surfaces are to be used for mounting as part of themetadata. Manual measurement may be done using devices such as rulers,tape measures and digital measuring devices such as laser distancemeasuring tools. When providing manual measurements, the end user mayspecify the length measured along with pixels in the digital imagecorresponding to the end points of the length measured. In oneembodiment, the user may use an ancillary object feature that demarcatesa target object measurement point for an end point of a manualmeasurement. The manual measurement may be confirmed by image processingand analysis methods described above. If the manual measurementsignificantly differs from the measurement estimated by image processingand analysis, feedback may be provided to the user to manuallyre-measure the target object dimension. Alternatively, the measurementestimated by image processing and analysis may be provided to the userprior to manual measurement or its input as metadata by the user.

Further, the end user may provide metadata about reference and/orancillary object dimensions in each image such as location and numericalvalue dimensions. Methods for facilitating location metadata input mayinclude zoom capability as is known in the art, which may be exemplifiedby software such as random access JPEG described in U.S. Pat. Nos.7,038,701, 7,652,595 and 7,847,711 to allow for location identificationusing the capture device, all of which are incorporated herein byreference in their entirety. Alternatively, the image may be transportedto a computer, uploaded to a website or transferred to a kiosk to allowthe user to point and click on the reference and/or ancillary objectsand enter information about the objects, including physical dimensionsor the location of measurement point demarcating features.

The methods described for correcting images of fenestration areparticularly useful when used to design custom supplemental parts havingmeans of adjustment or conformable deformation when compressed withinthe constraint surface dimensions calculated. Deformation means may beincorporated into the custom supplemental parts through the use ofcontinuously deformable means, for example, cantilevers, compressiblefoam, for example a polymer foam, or tube, or piles. Such conformablecompression means may also be used in conjunction with continuous ornon-continuous adjustment means such as a snap fit means. Thecompressible and adjustment means may be used to provide compression fitto more than one depth location of the window frame since there arerelatively small differences in the window frame dimensions at differentdepths within the window frame. Thus, a single set of customsupplemental parts may be used with different constraint surfaces.

In another embodiment of the present invention, measurements from enduser provided images may be corrected using lookup tables and camerametadata. The lookup tables may contain camera specific informationabout distortions (e.g., optical distortions such as lens relateddistortions) that could lead to measurement errors, including barrel,pincushion or complex distortions. The lookup tables may be based onprevious calibration studies for each particular camera.

With image and associated metadata information, relevant constraintsurface dimensions are calculated. The calculation of lengths may bedone automatically for all possible products and surfaces or may belimited to those selected by the end user/designee provided metadata.Lengths may be automatically calculated based on image informationshowing consistent continuous surfaces for mounting. Alternatively, asemi-automated method may be used in which such surfaces may beidentified from metadata provided by end user/designee or by the serviceprovider with human intervention.

With calculated lengths available from measurement algorithms describedsupra, custom supplemental parts are then fabricated. Using metadataprovided by the end user or designee, appropriate materials are used,cut to size, imprinted with relevant information and packaged. Forexample, the end user or designee may specify among several options suchas overlay or in-frame, adhesive or pressure mounting, location ofadhesive mount if chosen, whether window or window blind operability isdesired, if multiple sheets are to be used how many and where they areto be mounted. Such metadata may have been provided prior to submissionof an order.

Alternatively, the end user or designee may wish to obtain a costestimate prior to submitting an order. In this case, very roughmeasurements made prior to any image distortion correction may be usedto estimate the materials needed for various supplemental parts so thatvarious options and their associated costs may be provided prior toorder submission. With this information, an order may be generated to acentralized fabrication site or multiple distributed fabrication sites.Centralized fabrication entails fabrication of all custom supplementalparts at a single site where the parts may also be assembled forpackaging and delivery to the end user. When distributed fabrication isused, each fabricator may fabricate a subset of the parts necessary forfull functionality of the delivered product. The subset parts may besent to an order collation site for packaging and/or assembly of finalproduct parts prior to shipping to the end user. To minimize materialwaste during fabrication, it may be desirable to compile multiple ordersfor each subset part to allow for an optimized fabrication run.

It will be appreciated that measurements made by the methods describedherein may also be useful for applications that do not lead tofabrication of parts. For example, if the target object in each of aplurality of images is a different wall of the same room, it is possibleto obtain the dimensions of a room for real estate, architectural,engineering, or any other purpose. Alternatively, by measuring adimension of a piece of furniture located remote to a room in which itis desired to place the piece of furniture, it is possible to determinewhether the piece of furniture will fit properly within a desired spacein the room using the present invention to measure room dimensions. Itwill also be appreciated that using multiple reference objects residingin different planes and substantially coplanar with target objects thatare parallel to the imaging plane may be used to measure multiple targetobjects that are captured in the same digital image.

Note that measurement of the fenestration or portions thereof may beperformed manually or specified and delivered using the methodsdescribed herein. These measurement methods may also be used to confirmthe accuracy of manual measurements that may be provided to the serviceprovider or fabricator. They also serve to provide feedback to themanual measurement provider regarding accuracy and optionally requestingre-measurement.

The automated measurement methods described herein may be performedusing colored objects consisting of a thin, flat piece of paper, plasticor other suitable material, preferably highly colored (e.g., brightyellow, orange, red or other colors) such that the thin, flat materialcontrasts with the fenestration surroundings and/or the window panebackground. The thin, flat material may or may not have a re-adherablestrip of low-tack or medium-tack pressure sensitive adhesive. Coloredpieces of paper without a re-adherable strip of adhesive, may be appliedto the scene to be captured using adhesive tape. Colored pieces of paperwith a re-adherable strip of adhesive may also be used. Such papers arecommonly known as “repositionable notes”, e.g., Post-it® Notesmanufactured by 3M Corporation, St. Paul, Minn., USA, sticky notes,self-stick notes, etc.). Note that preferably, the colored pieces ofpaper are either square or rectangular shaped but are not limitedthereto.

Such colored papers are used, for example, as (1) any type of ancillaryobject including (a) pane ancillary objects to identify target objectlocations (e.g., edges or corners of the pane or sash); (b) sealinginterface ancillary objects for identifying the sealing interfaces suchas between the sash and the frame; (c) non-transparent target objectancillary objects, such as (i) frame ancillary objects for aiding inidentifying window frame edges for measurements, such as inward facingframe or jamb edges or outward facing frame casing edges or (ii) wallancillary objects for aiding in identifying wall edges or portions of awall for measurement; (2) adhering objects for adhering the referenceobject to the window or non-transparent target object to be measured;and (3) contrast providing objects (or simply contrast objects) forproviding high contrast between at least one reference object edge andat least one of a non-transparent target object to be measured and asecondary object in the captured digital image. In addition oralternatively to color and shape, ancillary objects, adhering objectsand/or contrast objects may have a pattern, for example a fiducialpattern, or entropy as a characteristic.

Note that all ancillary objects used in a scene are preferably the samecolor. Similarly, it is preferable that all adhering objects used in ascene be the same color. The contrast objects in a scene, however, maybe the same or different color. Placement location instructions foradhering objects or contrast objects on the reference object may beprovided by a service provider to the end user. By following suchinstructions, the locations of adhering or contrast objects may be moreeasily found in the automated measurement methods described.

When using a pane ancillary object to identify an edge or a corner, itis preferable to position the re-adherable strip of the pane ancillaryobject such that it is adjacent to an edge or corner to be identified bythe pane ancillary object with less than about one eighth inch,preferably less than about one sixteenth inch, between the edges of there-adherable strip and the edge or corner to be identified. Whenpositioned at corners or sash edges of the window pane, the paneancillary object functions to aid in distinguishing the pane edge fromother edges that may appear in the pane area of the image, such asreflections, storm windows or other sash edges that may exist to theexterior of the interior pane surface.

A diagram illustrating an example window with reference object, adheringobjects and pane ancillary objects is shown in FIG. 21. The window,generally referenced 450, comprises a frame casing 454 embedded in awall 452, top sash 457, top sash window pane 456, muntins 458, bottomsash 460 and bottom sash window pane 462. A reference object 464 isattached to the window via one or more adhering objects 468. One or morepane ancillary objects 466 are attached to the window pane in diagonallyopposite corners. Note that the location of adhering objects 468 shownare appropriate for non-protruding muntins. When protruding muntins arepresent, it is preferable to affix each adhering object to a muntin.

For example, when positioned by a user in diagonal pane corners, asshown in FIG. 21, the corner and edges of the pane ancillary objectnearest each pane corner may be used to aid in the determination of theperspective transform applied to the captured image. When using objects,such as pieces of paper, having the same size and color for (1) theadhering object to adhere the reference object to the window (such asthe pane or muntins) or other non-transparent target objects to bemeasured; and for (2) the pane ancillary object, the pixel scale factorobtained from the reference object may be used to determine a dimensionof the colored pieces of paper, preferably the dimension along the sidehaving the pressure sensitive adhesive. The preferred dimension alongthe side having the pressure sensitive adhesive, which is generally astrip about 0.625 inches wide, is in the range of approximately 1 to 4inches, more preferably about 1.5 to 3 inches.

A diagram illustrating an example window with reference object, adheringobjects, pane ancillary objects and sealing interface or inward facingframe and sill ancillary objects is shown in FIG. 22. The window,generally referenced 470, comprises a frame casing 474 embedded in awall 472, top sash 477, top sash window pane 476, muntins 478, bottomsash 480 and bottom sash window pane 482. A reference object 484 isattached to the window via one or more adhering objects 489. Two or morepane ancillary objects 486 are attached to the window pane in diagonallyopposite corners. One or more sealing interface ancillary objects areattached to the sash to aid in determining the location of the sealinginterface between the sash and the frame.

When inward facing frame ancillary objects are used, such objects areplaced with the adhesive strip along the interface of the sash orinterior facing frame and the inward facing frame. While the non-paneancillary objects are shown at sealing interface or inward facing framelocations, such ancillary objects may also be used to identify theoutward frame edges by adhering such ancillary objects to the interiorsurface of frame 477 with the adhesive strip along the outward edge ofthe frame or sill.

To find the sealing interface locations identified by sealing interfaceancillary objects in FIG. 22, the sealing interface ancillary objects488 are located in the image. In one embodiment, the sealing interfaceancillary objects are found using the methods described herein forlocating ancillary objects (i.e. the method of FIG. 25 described infra).In this embodiment, the sealing interface ancillary objects 488 may bedistinguished from the pane ancillary object 486 and optional paneancillary object 471 by their location with respect to the pane and/orthe use of user provided metadata (e.g., touch or click locations) toidentify the different types of ancillary objects. Optional paneancillary object 471 on the upper pane may aid in determining the heightof the checkrail between the lower and upper panes and/or may aid indetermining the location of the upper sash stile to pane edge locationto provide more accurate upper pane width measurement. When used solelyfor determining the checkrail height, pane ancillary object 471 may beplaced at any location along the checkrail to upper pane edge. When usedto locate the upper pane stile to pane edge, pane ancillary object 471is placed at the corner of the upper pane as shown in FIG. 22.

In the transformed image, the pixel distance from pane edge parallel toand nearest the sealing interface ancillary object bounding boxoutwardmost edge is determined and converted to the physical dimensionof the sash width. Symmetry may be used to determine additional sealinginterface locations. In addition, the placement of similar colored paperancillary objects can be used to determine inside mounting dimensionsfor window treatments.

A diagram illustrating an example window with reference object on theframe portion of the window, adhering objects, contrast providingobjects, ancillary objects and sealing interface ancillary objects isshown in FIG. 23. The window, generally referenced 490, comprises aframe casing 494 embedded in a wall 492, top sash 497, top sash windowpane 496, muntins 498, bottom sash 500 and bottom sash window pane 502.A reference object 506 is attached to the frame casing 494 via one ormore adhering objects 509. One or more contrast providing objects 508are attached to the back of the reference object (preferably straddlingthe corners or straddling an edge) and function to provide backgroundcontrast between the reference object and the target object and/orsecondary object such as the surrounding wall. One or more paneancillary objects 505 are optionally attached to the window pane indiagonally opposite corners. One or more frame ancillary objects 504 areattached to the window frame to aid in determining the location ofand/or measuring the frame.

A diagram illustrating an example window with reference object on anon-transparent target object, adhering objects, contrast providingobjects, ancillary objects and sealing interface ancillary objects isshown in FIG. 24. The window, generally referenced 510, comprises aframe casing 514 embedded in a wall 512, top sash 517, top sash windowpane 516, muntins 518, bottom sash 520 and bottom sash window pane 522.A reference object 528 is attached to the non-transparent target object511 (e.g., the wall in this example) via one or more adhering objects530. One or more contrast providing objects 532 are attached to the backof the reference object (preferably in the corners or straddling anedge) and function to provide background contrast between the referenceobject and the target object. One or more pane ancillary objects 526 areoptionally attached to the window pane in diagonally opposite cornersfor estimation of window pane dimensions as described supra. One or moreframe ancillary objects 524 are optionally attached to the window frameto aid in determining the location of and/or measuring the frame asdescribed supra. Non-transparent target objects may include any regular(i.e. rectangular or circular) or irregular (i.e. arbitrary) shapedobjects such as walls, objects hanging on walls, furniture, pictures,holes or openings in walls, etc.

Note that in one embodiment, one or more edges of wall 511 or a portionthereof may be defined with one or more ancillary objects 523. Such ascene may be used for estimating the dimensions and/or area of wall 511or a portion thereof. Such scenes may be useful, for example, forestimating the amount of paint or wallpaper needed to cover the wall or,if the wall requires repair, the area and/or dimensions to be repaired.As an example, a repair may be required to patch a hole 525 in the wall.In this example, multiple ancillary objects 527 (two upper and twolower) may be placed around the perimeter, such as at the corners oredges, of the hole to aid in estimating the dimensions and/or area ofthe hole.

FIGS. 23 and 24 illustrate additional uses of colored pieces of papersuch as self-adhering notes in the present invention. To aid indetermining the reference object pixel scale factor and measuringreference object pixel dimensions, such colored pieces of paper may beused to provide background contrast for the reference object in theevent it has low contrast against the non-transparent target object tobe measured, (e.g., a sheet of white copier paper against a white orcream colored wall). The colored pieces of paper referred to as contrastproviding objects (e.g., self-adhering notes) are placed on the back ofthe reference object along edges or corners with a portion of thecolored piece of paper visible immediately adjacent the edge or cornerof the reference object. When used in this manner, mounting of thereference object may be made on non-transparent portions of thefenestration (FIG. 23), next to the fenestration or on anynon-transparent target object or surface such as a wall (FIG. 24).

For example, adhering the reference object 506 to the window frame 494as shown in FIG. 23 is useful when measuring the dimensions of thewindow frame. Adhering the reference object as shown in FIG. 24 isuseful when measuring the dimensions 521 of the wall 511. In eithercase, pane ancillary objects adhered at diagonal window pane corners maybe used to aid with the perspective transform, while frame casingancillary objects adhered to the window frame aid in identifying windowframe edges for measurements. Alternatively, reference object 528, orfeatures on the reference object such as a printed pattern of knownaspect ratio, may be used for determining the perspective transform,particularly in cases where no window is present on the wall to bemeasured. For either of these reference object locations, depth offsetcorrection may be used as described supra for measuring target objectdimensions that are slightly offset from the plane to which thereference object is adhered. Such depth offset corrections may beobtained using the methods described supra, incorporating the differencein pixel dimensions found for pane ancillary objects and adheringobjects which are the same physical size and that reside in the twosubstantially parallel planes. Thus, adhering the reference object asshown in FIG. 24 may also be useful for measuring window elements on thesame wall as the reference object.

As shown in FIG. 23, the adhering objects 509 may also perform thefunction of frame ancillary objects for identifying outward frame edgelocations. In this case the additional frame ancillary objectidentifying the same outward frame edge is not needed. Preferably, thecontrast providing object 508 has sufficiently high contrast and/orcolor difference with respect to the target object (e.g., window frameand sashes) and secondary objects (e.g., the wall surrounding the targetobject and any window treatments that may be present). The contrastproviding objects may be the same or different in color and/or size withrespect to the adhering objects used to adhere the reference object toan object in the scene. An alternative use of such contrasting pieces ofcolored paper to provide reference object background contrast is in themeasurement of non-transparent target objects having a substantiallyplanar surface such as, for example, walls or portions thereof,cabinetry, furniture or appliances.

When using contrast providing objects to aid in the identification ofreference object edge locations, the methods described herein forfinding and measuring the reference object pixel scale factor andrelated dimensions may be used, where the edges defined by the contrastproviding objects are used.

Note that the locations of the colored pieces of paper (e.g.,repositionable notes) may be found using (1) an image processingalgorithm as described in more detail infra; or (2) a combination of (a)an image processing algorithm and (b) user supplied metadata such aspixel locations corresponding to the approximate or exact location of anobject of interest (e.g., reference object, pane ancillary object,target object feature, etc.) that is identified by touching, tapping orclicking on the digital image when displayed on a device capable ofcapturing touched, tapped or clicked pixel locations.

Well-known image processing algorithms or tools generally known as imagesegmentation algorithms may be used to find objects having similarcontiguous characteristics such as color, e.g., k-means clustering orother color based image segmentation methods. In one embodiment, asuitable tool for use with the invention is the magic wand tool suppliedin Photoshop or Photoshop Elements commercial software applicationsavailable from Adobe Systems Incorporated, San Jose, Calif., USA.

In an automated or semi-automated embodiment, functions such as thoseavailable in OpenCV: cvInrangeS or cvThreshold may be used to thresholdan image for color attributes such as hue, saturation or brightness.When using functions for finding specific colors in an image, the imageis generally first converted as needed to a suitable color space, forexample, RGB, CMYK, Y′CrCb or any other suitable color space, preferablyan HSV or a Lab type color space. Thresholding or filtering may then beperformed using upper and lower bound for values such as hue, luminanceand/or chroma that include the color of the colored object. When usingcolored re-positionable notes of the same color to both adhere thereference object to the fenestration (i.e. adhering objects) and as paneancillary objects, estimation of the bounds for thresholding orfiltering may be aided by first determining the color space values ofthe objects used to adhere the reference object to the fenestration.Portions of the image not immediately adjacent to the reference objectmay then be thresholded or filtered for color with similar color spacevalues as those on the adhering objects to identify the location ofcolored pane ancillary objects. This may be followed by applying a blobfinder, for example, using the detect function of thecv::SimpleBlobDetector class to provide colored ancillary objectlocations. Other OpenCV functions that may be useful includecv2.watershed, cv.findContours and cv.morphologyEx. In one embodiment,the method described athttp://stackoverflow.com/questions/11294859/how-to-define-the-markers-for-watershed-in-opencv,incorporated herein by reference in its entirety, may be used in which agrayscale image defined using the distance from a known color using, forexample, the distance metric in an HSV or Lab type color space.

For example, the watershed function may be used to separate similarobjects rather than grouping them into a single component. Markers areused for each object and the background. The input image may bebinarized, for example using well-known Otsu's method for performingclustering-based image thresholding or reducing a gray level image to abinary one, and performing morphological opening. A distance transformmay be applied to the binary image followed by application of athreshold to provide regions most distant from the background. A markerfor each object is obtained by labeling different regions. Each markeris associated with a “colored water” to fill the regions between theobjects and the watershed transformation constructs dams to impedemerging of the different “colors”. The dams composed on the originalimage then form outlines around the objects.

Methods for measuring colors and determining their similarities anddifferences may include any such methods known in the art, includingDelta CIELAB and Delta CIELCH as well as tolerancing methods in CIELAB,CIELCH, CMC or CIE94, may use spherical or ellipsoidal distances as wellas asymmetrical weighting of lightness and hue, as described in, forexample, “A Guide to Understanding Color Communication” which can befound on the Internet athttp://www.xrite.com/documents/literature/en/L10-001_Understand_Color_en.pdf,incorporated herein by reference in its entirety.

A flow diagram illustrating an example method for finding ancillaryobjects such as self-adhesive notes is shown in FIG. 25. This method maybe used to find one or more ancillary objects after placement of thepane ancillary objects on or near the window. The number and location ofancillary objects may vary such that various steps of this method areoptional and may be included depending on the particular implementationof the invention.

To find colored paper ancillary objects in an uncorrected or correctedimage, the reference object is first found (step 540) using (1) methodsdescribed supra (e.g., the method of FIG. 9) using a fiducial 503 (FIG.23) printed on the reference object; or (2) metadata such as one or moreimage locations identified by the end user or service provider touching,tapping or clicking on the reference object to provide one or moreregions of interest for locating the reference object.

Once the reference object is located, the adhering objects are found inthe digital image by color and/or shape segmentation of the image (step541). An adhering object characteristic (e.g., color, shape, etc.) isthen selected (step 542). For example, using k-means, the color of theadhering object may be determined. The color is then used to segment theimage based on the selected characteristic (e.g., color, shape, etc.)(step 544) to identify one or more ancillary objects (e.g.,self-adhesive notes not adhered to the reference object), such as thoseshown in FIGS. 21 and 22 (step 546). The segmented color not overlappingwith the reference object represents the ancillary objects and boundingboxes 467, 469 (FIG. 21) are respectively placed around each found paneancillary object (step 548). Optionally, end user supplied metadataindicating locations in the digital image may be used with the methodillustrated in FIG. 25. Such metadata may be used to provide regions ofinterest in which to seek the ancillary object selected characteristic(e.g., color). When locating pane ancillary objects, the methodillustrated in FIG. 12 may optionally be used to provide a pane boundingbox, the edges or corners of which may be used as regions of interest inwhich to seek the pane ancillary object selected characteristic.

A flow diagram illustrating an example method for determiningperspective transform is shown in FIG. 26. The bounding boxes 467, 469(FIG. 21) previously found using the method of FIG. 25 are used todefine horizontal and vertical band regions of interest near therectangle 465 defined by the corners of the bounding boxes and thevertical and horizontal bands (step 550). An edge detector, such as thewell-known Canny edge detector and probabilistic Hough transform may beused to create a consensus line within each region of interest (step552). The distorted rectangle formed by the consensus lines provide thesource points used to generate a preliminary perspective transform thatwill result in a rectangle (step 554). Destination points for thepreliminary perspective transform are created using the midpoints ofeach of the edge lines as described supra (step 556). Optionally, theslopes of the consensus lines could be used to generate lines goingthrough the corner points at the pane corners. The resulting transformis designed to map the respective source corners exactly onto thedestination corners (step 558). Once the preliminary perspectivetransform is obtained, it is adjusted so that the center point andcenter geometric mean scale of the original image is preserved such thatthe aspect ratio of the reference object is maintained. This is achievedby balancing the relative scaling of the 2D transform (step 560),normalizing the perspectivity matrix (step 562) and preserving the imagecenter location (step 564).

A flow diagram illustrating an example method for calculating targetobject dimensions is shown in FIG. 27. The self-adhering pane ancillaryobjects are re-found in the corrected/transformed image (step 570). Inone embodiment, this is performed by using their saved locations in thedistorted image. A bounding box 465 is drawn that includes theself-adhering pane ancillary object corner points 463, 461 nearest theprimary object corners or edges that the ancillary object identifies(step 572). Thus, for ancillary objects placed at diagonal corners of awindow pane such as shown in FIG. 21, the resulting bounding box 465defines the pixel locations of the pane-sash edges. The pixel dimensionsof the transformed bounding box 465 created from the pane ancillaryobjects are then determined (step 574). The pixel dimensions are thenconverted to the pane physical dimensions using the reference objectpixel scale factor (i.e. pixels per unit length) (step 576).

The method for finding ancillary objects illustrated in FIG. 25 may alsobe used to find ancillary objects used to define non-transparent targetobjects such as walls or portions thereof. In such cases, an ancillaryobject edge adhered to the wall may be used to define a wall corner, theedge defined where the wall and floor or the wall and ceiling meet. Analternative, semi-automated, method may be preferred, especially fordetermining the wall/ceiling edge, in which ancillary objects are usedfor easily reached edges and a software tool, for example a cropping,marquee or line tool, is used by the end user, service provider orfabricator to identify the ceiling to wall edge in the digital image.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. As numerousmodifications and changes will readily occur to those skilled in theart, it is intended that the invention not be limited to the limitednumber of embodiments described herein. Accordingly, it will beappreciated that all suitable variations, modifications and equivalentsmay be resorted to, falling within the spirit and scope of the presentinvention. The embodiments were chosen and described in order to bestexplain the principles of the invention and the practical application,and to enable others of ordinary skill in the art to understand theinvention for various embodiments with various modifications as aresuited to the particular use contemplated.

What is claimed is:
 1. A method of determining a dimension of a targetobject in a digital image implemented by a system comprising one or morecomputing devices, the method comprising: obtaining a digital imagehaving the target object and one or more ancillary objects locatedtherein from an image capture device, the one or more ancillary objectslocated in a scene of the digital image to define target objectmeasurement points of the target object; determining a distance from aplurality of objects in the scene of the digital image to the imagecapture device; creating a three-dimensional model of the plurality ofobjects in the scene of the digital image based on the determineddistance; locating the target object measurement points in thethree-dimensional model; calculating a distance between the targetobject measurement points; and determining the dimension of the targetobject based on the calculated distance between the target objectmeasurement points.
 2. The method according to claim 1, wherein thedigital image scene contains a transparent object and the transparentobject is an element of the target object or is associated with thetarget object.
 3. The method according to claim 2, wherein thetransparent object is a window and the target object is a window elementor associated with the window.
 4. The method according to claim 1,wherein the locating is performed using a distinctive feature of theancillary object.
 5. The method according to claim 4, wherein thedistinctive feature of the ancillary object is one of a color or afiducial pattern.
 6. The method according to claim 1, wherein thelocating is performed using metadata.
 7. The method according to claim6, wherein the metadata is obtained from user input.
 8. The methodaccording to claim 6, wherein the metadata is obtained from a digitalapplication.
 9. The method according to claim 3, wherein the windowelement is a sash, stile, frame, sill, jamb, jamb liner or pane of thewindow.
 10. The method according to claim 2, wherein the transparentobject is reflective.
 11. The method according to claim 2, wherein thetransparent object appears dark in the digital image.
 12. The methodaccording to claim 2, wherein the transparent object is captured underconditions of predominantly back lighting.
 13. The method according toclaim 1, wherein the digital image comprises at least a visible imageand an infrared image images.
 14. The method according to claim 1,wherein the distance between the target object measurement points iscalculated using the three-dimensional model.
 15. The method accordingto claim 1, wherein the distance between the target object measurementpoints is calculated using a pixel scale factor.
 16. The methodaccording to claim 2, wherein the ancillary objects are placed on thetransparent object.